-Can't HDFS from within streaming mapper
Keith Wiley 2012-01-10, 20:00
My streaming mapper does and HDFS "get" to pull a file to the local node. This has worked perfectly on another cluster, but it's timing out on a new cluster:
Command: 'hadoop fs -get hdfs://mainclusternn.hipods.ihost.com/path/file.gz file.gz'
get: Call to mainclusternn.hipods.ihost.com/<ip>:8020 failed on socket timeout exception: java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=mainclusternn.hipods.ihost.com/<ip>:8020]
However, I can easily see the file from the command line:
$ hadoop fs -ls /path/file.gz
Found 1 items
-rw-r--r-- 1 kbwiley supergroup 2275203 2012-01-09 16:18 /path/file.gz
I'm unsure how to proceed.
Keith Wiley [EMAIL PROTECTED] keithwiley.com music.keithwiley.com
"The easy confidence with which I know another man's religion is folly teaches
me to suspect that my own is also."
-- Mark Twain