Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> Exception while syncing from Flume to HDFS


+
Divya R 2013-07-19, 07:20
Copy link to this message
-
Re: Exception while syncing from Flume to HDFS
hi,

this is not HDFS issue.
you can put your question in the flume ml.
 On Jul 19, 2013 3:20 PM, "Divya R" <[EMAIL PROTECTED]> wrote:

> I'm running hadoop 1.2.0 and flume 1.3. Every thing works fine if its
> independently run. When I start my tomcat I get the below exception after
> some time.
>
>   2013-07-17 12:40:35,640 (ResponseProcessor for block
> blk_5249456272858461891_436734) [WARN -
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:3015)]
> DFSOutputStream ResponseProcessor exception  for block
> blk_5249456272858461891_436734java.net.SocketTimeoutException: 63000
> millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/127.0.0.1:24433
> remote=/127.0.0.1:50010]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readLong(DataInputStream.java:416)
> at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2967)
>
>      2013-07-17 12:40:35,800 (hdfs-hdfs-write-roll-timer-0) [WARN -
> org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:277)]
> failed to close() HDFSWriter for file
>
> (hdfs://localhost:9000/flume/Broadsoft_App2/20130717/jboss/Broadsoft_App2.1374044838498.tmp).
> Exception follows.
> java.io.IOException: All datanodes 127.0.0.1:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3096)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100(DFSClient.java:2589)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2793)
>
>
>   Java snippet for Configuraion
>
> configuration.set("fs.default.name", "hdfs://localhost:9000");
> configuration.set("mapred.job.tracker", "hdfs://localhost:9000");
>
> I'm using a single datanode to read the files that where written to hdfs by
> flume, my java program just reads the files from hdfs to show it on the
> screen nothing much. Any sort of help is highly appreciated.
>
>
> Regards,
> Divya
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB