Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: Exception in createBlockOutputStream - poss firewall issue


Copy link to this message
-
Re: Exception in createBlockOutputStream - poss firewall issue
Clients will read/write data to the DNs directly. DNs serve on port 50010
and 50020 by default. Please open up these ports, aside of the NN's RPC
ports, to be able to read/write data.
On Thu, Jun 27, 2013 at 2:23 AM, Robin East <[EMAIL PROTECTED]> wrote:

> I have a single node hadoop cluster setup behind a firewall and am trying
> to create files using a java program outside the firewall and get the
> exception below. The java program works fine inside the firewall. The ip
> address for the single cluster is xxx.xxx.xxx.xxx however it appears that
> in the createBlockOutputStream the client things the data node is at ip
> yyy.yyy.yyy.yyy (the internal address of the cluster) which is not
> accessible.
> The java code looks like this (using hadoop 1.1.2):
>
>     private static void createHdfsFile() throws IOException {
>         Configuration conf = new Configuration();
>         conf.set("fs.default.name", "hdfs://"+hdfsHost+":9000");
>         FileSystem hdfs = FileSystem.get(conf);
>         System.out.println("HDFS Working Directory: " +
> hdfs.getWorkingDirectory().toString());
>         FSDataOutputStream os = hdfs.create(new
> Path("/user/hadoop/test2.txt"));
>
>         os.writeChars("Example text\n for a hadoop write
> call\n\ntesting\n");
>         os.close();
>     }
>
> Any idea how I can get this to work?
>
>
> HDFS Working Directory: hdfs://xxx.xxx.xxx.xxx:9000/user/zzzzz
> Jun 26, 2013 8:08:52 PM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream
> createBlockOutputStream
> INFO: Exception in createBlockOutputStream yyy.yyy.yyy.yyy:50010
> java.net.ConnectException: Connection timed out: no furth
> er information
> Jun 26, 2013 8:08:52 PM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream
> nextBlockOutputStream
> INFO: Abandoning block blk_4933973859208379842_1028
> Jun 26, 2013 8:08:53 PM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream
> nextBlockOutputStream
> INFO: Excluding datanode yyy.yyy.yyy.yyy:50010
> Jun 26, 2013 8:08:53 PM
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer run
> WARNING: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: File /user/hadoop/test2.txt
>  could only be replicated to 0 nodes, instead of 1
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>         at $Proxy1.addBlock(Unknown Source)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>         at $Proxy1.addBlock(Unknown Source)
Harsh J
+
Robin East 2013-06-27, 06:17
+
John Lilley 2013-07-02, 18:48