Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Can't HDFS from within streaming mapper


Copy link to this message
-
Can't HDFS from within streaming mapper
My streaming mapper does and HDFS "get" to pull a file to the local node.  This has worked perfectly on another cluster, but it's timing out on a new cluster:

Command: 'hadoop fs -get hdfs://mainclusternn.hipods.ihost.com/path/file.gz file.gz'
get: Call to mainclusternn.hipods.ihost.com/<ip>:8020 failed on socket timeout exception: java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=mainclusternn.hipods.ihost.com/<ip>:8020]

However, I can easily see the file from the command line:

$ hadoop fs -ls /path/file.gz                            
Found 1 items
-rw-r--r--   1 kbwiley supergroup    2275203 2012-01-09 16:18 /path/file.gz

I'm unsure how to proceed.

________________________________________________________________________________
Keith Wiley     [EMAIL PROTECTED]     keithwiley.com    music.keithwiley.com

"The easy confidence with which I know another man's religion is folly teaches
me to suspect that my own is also."
                                           --  Mark Twain
________________________________________________________________________________
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB