Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Can't HDFS from within streaming mapper

Copy link to this message
Can't HDFS from within streaming mapper
My streaming mapper does and HDFS "get" to pull a file to the local node.  This has worked perfectly on another cluster, but it's timing out on a new cluster:

Command: 'hadoop fs -get hdfs://mainclusternn.hipods.ihost.com/path/file.gz file.gz'
get: Call to mainclusternn.hipods.ihost.com/<ip>:8020 failed on socket timeout exception: java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=mainclusternn.hipods.ihost.com/<ip>:8020]

However, I can easily see the file from the command line:

$ hadoop fs -ls /path/file.gz                            
Found 1 items
-rw-r--r--   1 kbwiley supergroup    2275203 2012-01-09 16:18 /path/file.gz

I'm unsure how to proceed.

Keith Wiley     [EMAIL PROTECTED]     keithwiley.com    music.keithwiley.com

"The easy confidence with which I know another man's religion is folly teaches
me to suspect that my own is also."
                                           --  Mark Twain