记一次hadoop Connection refused: no further information排错

前言

music:

border="0" width="330" height="86" src="//music.163.com/outchain/player?type=2&id=139774&auto=1&height=66">

这两天在研究hadoop的hdfs,用本地java api远程调用hdfs时出现了连接被拒绝的问题,具体异常如下:

Caused by: java.net.ConnectException: Connection refused: no further information
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:687)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:790)
    at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:411)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1554)
    at org.apache.hadoop.ipc.Client.call(Client.java:1385)
    ... 48 more

异常如下图:
这里写图片描述
记录一下解决方案,希望可以帮到和我遇到相同问题的人。

客户端代码调用

package com.sy.hadoop;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import java.net.URI;

/**
 * @author suyu
 * @version 1.0.0
 * @ClassName HadoopApp
 * @Description Java Api操作Hadoop的 HDFS 文件
 * @Date 2018/7/28 16:15
 */
public class HadoopApp {

    FileSystem fileSystem = null;
    Configuration configuration = null;

    /**
     * HDFS的路径,core-site.xml中配置的端口号
     */
    public static final String HDFS_PATH = "hdfs://192.168.1.102:8082";
    /**
     * 解决无权限访问,设置远程hadoop的linux用户名称
     */
    public static final String USER = "weblogic";

    /**
     * 单元测试之前的准备工作,准备环境,加载配置
     *
     * @throws Exception
     */
    @Before
    public void setUp() throws Exception {
        System.out.println("HDFSApp is setUp.....");
        configuration = new Configuration();
        fileSystem = FileSystem.get(new URI(HDFS_PATH), configuration, USER);

    }

    /**
     * 创建目录
     *
     * @throws Exception
     */
    @Test
    public void mkDir() throws Exception {
        fileSystem.mkdirs(new Path("/hdfsapi/test"));
    }

    /**
     * 单元测试之后的工作,清理环境,释放资源等等
     *
     * @throws Exception
     */
    @After
    public void tearDown() throws Exception {
        configuration = null;
        fileSystem = null;
        System.out.println("HDFSApp is tearDown....");
    }
}

在调用单元测试的时候,想模拟创建一下目录,结果发生异常!

解决方案

下面的排错步骤,是建立在hadoop的服务端防火墙关闭的情况下(这点注意一下!)

第一步

1.查看你的hadoop Api提供的地址 + 端口!
如何查看正确的地址呢?
打开你的hadoop家目录!然后进入etc/hadoop比如我的就是:

/hadoop/hadoop-2.9.0/etc/hadoop

然后查看core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:8020</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/hadoop/tmp</value>
    </property>
</configuration>

其中第一个property的fs.defalutFS节点下的value,便是你本地api调用的地址加端口!

第二步

2.在hadoop的服务端调用如下命令,检查是否打开:

telnet 192.168.1.102 8020

若看到如下文本,说明服务端是没有问题的!

Trying 192.168.1.102...
Connected to test (192.168.1.102).
Escape character is '^]'.
Connection closed by foreign host.

在本地的windows端,检查客户端是否可以调通服务端ip端口:
tips:windows下的cmd命令中,调用telnet 是需要单独开启服务端,默认系统telnet命令在cmd中是调用是找不到的!
打开windows—–telnet服务链接教程:https://jingyan.baidu.com/article/3ea51489ba79e252e61bba97.html

telnet 192.168.1.102 8020

执行完毕,如果你看到的是跳入一个黑框,什么内容也没有,说明客户端也是可以调通服务端的!!!

第三步

3.查看你java代码本地的API中连接远程服务的ip以及端口是否正确HDFS_PATH(我代码中的变量)

先说我的第一步,就已经自己把自己坑了一次,我写的ip地址是对的但是端口号是错的!!!
因为是手打的,所以不小心看成了8082…..(一定要心细啊!程序猿!!!~~)

"hdfs://192.168.1.102:8082";

好了,修改完这一步现在是正确的了!

    /**
     * HDFS的路径,core-site.xml中配置的端口号
     */
    public static final String HDFS_PATH = "hdfs://192.168.1.102:8020";

本以为可以成功,然而事实是残酷的!! 依然报出Connection refused: no further information。

继续排错!!

第四步

想了想,为什么会网络被拒绝呢!无非就是找不到服务端的ip地址!!为什么找不到呢!!!
再来看下hadoop服务端的核心配置文件!

cd /hadoop/hadoop-2.9.0/etc/hadoop

然后查看core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:8020</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/hadoop/tmp</value>
    </property>
</configuration>

配置文件中,我采用的是localhost的形式!!!问题就出现在这里了,如果对外提供服务,写成localhost,外部请求是不识别的!

修改其核心配置文件,将localhost改为ip地址:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://192.168.1.102:8020</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/hadoop/tmp</value>
    </property>
</configuration>

再试一次!成功如下:
这里写图片描述

去服务端用命令查看下是否创建成功:

#可以迭代看到hadoop根目录的文件夹
 hadoop fs -ls -R /

这里写图片描述

可以看到上图已经有刚刚创建的hdfsapi/test目录了!

总结

哎,总之程序员这个行业真是需要细心!否则会各种被坑。。。当然本次的排错思路也少不了hadoop官方wiki,人家写的更加权威一些。链接如下:
https://wiki.apache.org/hadoop/ConnectionRefused

已标记关键词 清除标记
server环境: 在virtualbox里安装了ubuntu,然后装了了hadoop2.7.5,hbase1.3.1,然后hadoop和hbase的环境都配置好了 启动了hbase,然后本机里通过scala访问虚拟机里的hbase时,connection能获取到,但是在执行tableExists时,卡一会后就报超时了,client的代码和server的配置如: server配置(其中zookeeper用的hbase自己管理的): hbase-site.xml ![图片说明](https://img-ask.csdn.net/upload/201804/02/1522641879_341754.png) client的代码: ``` class HbaseUtil { def GetHbaseConfiguration(ip:String):Unit={ var conf = HBaseConfiguration.create; //这个是远程hbase的ip地址 conf.set("hbase.zookeeper.quorum",ip) //conf.set("zookeeper.znode.parent", "/hbase-unsecure") //2181是hbase里zookeeper的默认端口号 conf.set("hbase.zookeeper.property.clientPort","2181") println(ip+":habse connention success...") val tableName = "jndata" conf.set(TableInputFormat.INPUT_TABLE, tableName) val con= ConnectionFactory.createConnection(conf) val hBaseAdmin = con.getAdmin//new HBaseAdmin(conf); val tn = TableName.valueOf(tableName); // hBaseAdmin.disableTable(tn) // println(tableName + " is exist,detele....") //创建表 if (hBaseAdmin.tableExists(tn)) { println(tableName + " is exist,....") }else{ println(tableName + " is not exist,....") } } } ``` 然后server端的zookeeper的log如下: ![图片说明](https://img-ask.csdn.net/upload/201804/02/1522640993_389812.png) client端的异常如下: ``` Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions: Mon Apr 02 11:30:11 CST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=76915: Connection refused: no further information row 'jndata,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=simon-virtualbox,16201,1522631406769, seqNum=0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289) at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:164) at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:159) at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:796) at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602) at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:408) at rxb.flinkDemo.hbase.HbaseUtil.GetHbaseConfiguration(HbaseUtil.scala:42) at rxb.flinkDemo.MyDemo$.main(FlinkDemo.scala:27) at rxb.flinkDemo.MyDemo.main(FlinkDemo.scala) Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=76915: Connection refused: no further information row 'jndata,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=simon-virtualbox,16201,1522631406769, seqNum=0 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:171) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused: no further information at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:416) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:722) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:909) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1244) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:35396) at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:404) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:211) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:65) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:137) ... 4 more ``` 好多办法都试过了还是不行: 1.server和client的防火墙都关了 2.client的host里也配置了server的ip和hostname
©️2020 CSDN 皮肤主题: 创作都市 设计师:CSDN官方博客 返回首页