Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Re: access to hadoop cluster to post tasks remotely


Copy link to this message
-
Re: access to hadoop cluster to post tasks remotely
Harsh J 2013-11-06, 23:48
Data in HDFS is read and written via the individual DN's 50010 ports,
which you would also need to open up to avoid these errors. Data isn't
written/read through the NameNode.

On Thu, Nov 7, 2013 at 4:50 AM, Sergey Gerasimov
<[EMAIL PROTECTED]> wrote:
> Hello,
>
>
>
> I have problems with posting jar to my cluster remotely from client machine
> located somewhere in the Web. I use original hadoop-1.2.1.
>
>
>
> I installed hadoop  on client machine (same version as in the cluster),
> configured fs.default.name and mapred.job.tracker.
>
> Access to DFS works fine remotely. I can successfully play with “hadoop fs”
> commands.
>
>
>
> But when I send some job, for example:
>
> hadoop jar hadoop-examples -1.2.1.jar sleep 1
>
>
>
> I see output like:
>
> 13/11/07 02:44:42 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out
>
> 13/11/07 02:44:42 INFO hdfs.DFSClient: Abandoning
> blk_1089181243677159149_31717
>
> 13/11/07 02:44:42 INFO hdfs.DFSClient: Excluding datanode xx.xx.xx.xx:50010
>
> 13/11/07 02:45:45 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out
>
> 13/11/07 02:45:45 INFO hdfs.DFSClient: Abandoning
> blk_6550586867464091073_31717
>
> 13/11/07 02:45:45 INFO hdfs.DFSClient: Excluding datanode xx.xx.xx.xx:50010
>
> 13/11/07 02:46:48 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out
>
> 13/11/07 02:46:48 INFO hdfs.DFSClient: Abandoning
> blk_5814098597599107248_31717
>
> 13/11/07 02:46:48 INFO hdfs.DFSClient: Excluding datanode xx.xx.xx.xx:50010
>
> 13/11/07 02:47:51 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out
>
> 13/11/07 02:47:51 INFO hdfs.DFSClient: Abandoning
> blk_6368219524592897749_31717
>
>
>
> The same jar sent from inside the cluster runs fine.
>
>
>
> The network where cluster lives  is protected by firewall with only NameNode
> and JobTracker ports opened externally.
>
> iptables on all nodes are off.
>
>
>
> I have no ideas about reasons of these messages in the log. To the moment I
> were sure that entry point to hadoop cluster  contains just NameNode and
> JobTracker ports.
>
> Both are open.
>
>
>
> Please help!
>
>
>
>
>
>

--
Harsh J