Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: access to hadoop cluster to post tasks remotely


Copy link to this message
-
RE: access to hadoop cluster to post tasks remotely
Oooops.

 

Not all "hadoop fs" commands works fine..

 

-ls is OK

-put/-get give similar error.

 

Looks like port 50010 of data nodes should be accessible externally.. Does
anybody know some config param to work around?

 

But I still don't understand why hadoop engine tries  to connect to
DataNodes from client(!) machine during posting jar from client machine to
the cluster.

 

From: Sergey Gerasimov [mailto:[EMAIL PROTECTED]]
Sent: Thursday, November 07, 2013 3:20 AM
To: [EMAIL PROTECTED]
Subject: access to hadoop cluster to post tasks remotely

 

Hello,

 

I have problems with posting jar to my cluster remotely from client machine
located somewhere in the Web. I use original hadoop-1.2.1.

 

I installed hadoop  on client machine (same version as in the cluster),
configured fs.default.name and mapred.job.tracker.

Access to DFS works fine remotely. I can successfully play with "hadoop fs"
commands.

 

But when I send some job, for example:

hadoop jar hadoop-examples -1.2.1.jar sleep 1

 

I see output like:

13/11/07 02:44:42 INFO hdfs.DFSClient: Exception in createBlockOutputStream
xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out

13/11/07 02:44:42 INFO hdfs.DFSClient: Abandoning
blk_1089181243677159149_31717

13/11/07 02:44:42 INFO hdfs.DFSClient: Excluding datanode xx.xx.xx.xx:50010

13/11/07 02:45:45 INFO hdfs.DFSClient: Exception in createBlockOutputStream
xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out

13/11/07 02:45:45 INFO hdfs.DFSClient: Abandoning
blk_6550586867464091073_31717

13/11/07 02:45:45 INFO hdfs.DFSClient: Excluding datanode xx.xx.xx.xx:50010

13/11/07 02:46:48 INFO hdfs.DFSClient: Exception in createBlockOutputStream
xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out

13/11/07 02:46:48 INFO hdfs.DFSClient: Abandoning
blk_5814098597599107248_31717

13/11/07 02:46:48 INFO hdfs.DFSClient: Excluding datanode xx.xx.xx.xx:50010

13/11/07 02:47:51 INFO hdfs.DFSClient: Exception in createBlockOutputStream
xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out

13/11/07 02:47:51 INFO hdfs.DFSClient: Abandoning
blk_6368219524592897749_31717

 

The same jar sent from inside the cluster runs fine.

 

The network where cluster lives  is protected by firewall with only NameNode
and JobTracker ports opened externally.

iptables on all nodes are off.

 

I have no ideas about reasons of these messages in the log. To the moment I
were sure that entry point to hadoop cluster  contains just NameNode and
JobTracker ports.

Both are open.

 

Please help!