Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: access to hadoop cluster to post tasks remotely


Copy link to this message
-
Re: access to hadoop cluster to post tasks remotely
Data in HDFS is read and written via the individual DN's 50010 ports,
which you would also need to open up to avoid these errors. Data isn't
written/read through the NameNode.

On Thu, Nov 7, 2013 at 4:50 AM, Sergey Gerasimov
<[EMAIL PROTECTED]> wrote:
> Hello,
>
>
>
> I have problems with posting jar to my cluster remotely from client machine
> located somewhere in the Web. I use original hadoop-1.2.1.
>
>
>
> I installed hadoop  on client machine (same version as in the cluster),
> configured fs.default.name and mapred.job.tracker.
>
> Access to DFS works fine remotely. I can successfully play with “hadoop fs”
> commands.
>
>
>
> But when I send some job, for example:
>
> hadoop jar hadoop-examples -1.2.1.jar sleep 1
>
>
>
> I see output like:
>
> 13/11/07 02:44:42 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out
>
> 13/11/07 02:44:42 INFO hdfs.DFSClient: Abandoning
> blk_1089181243677159149_31717
>
> 13/11/07 02:44:42 INFO hdfs.DFSClient: Excluding datanode xx.xx.xx.xx:50010
>
> 13/11/07 02:45:45 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out
>
> 13/11/07 02:45:45 INFO hdfs.DFSClient: Abandoning
> blk_6550586867464091073_31717
>
> 13/11/07 02:45:45 INFO hdfs.DFSClient: Excluding datanode xx.xx.xx.xx:50010
>
> 13/11/07 02:46:48 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out
>
> 13/11/07 02:46:48 INFO hdfs.DFSClient: Abandoning
> blk_5814098597599107248_31717
>
> 13/11/07 02:46:48 INFO hdfs.DFSClient: Excluding datanode xx.xx.xx.xx:50010
>
> 13/11/07 02:47:51 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> xx.xx.xx.xx:50010 java.net.ConnectException: Connection timed out
>
> 13/11/07 02:47:51 INFO hdfs.DFSClient: Abandoning
> blk_6368219524592897749_31717
>
>
>
> The same jar sent from inside the cluster runs fine.
>
>
>
> The network where cluster lives  is protected by firewall with only NameNode
> and JobTracker ports opened externally.
>
> iptables on all nodes are off.
>
>
>
> I have no ideas about reasons of these messages in the log. To the moment I
> were sure that entry point to hadoop cluster  contains just NameNode and
> JobTracker ports.
>
> Both are open.
>
>
>
> Please help!
>
>
>
>
>
>

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB