Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Error while writing to HDFS


+
Cameron Gandevia 2012-11-06, 02:43
+
Brock Noland 2012-11-06, 13:00
Copy link to this message
-
Re: Error while writing to HDFS
Thanks for the reply, it looks like the cause was our DataNodes throwing
the following exception

java.io.IOException: xceiverCount 2050 exceeds the limit of concurrent
xcievers 2048

I upgraded this setting and now everything seems to run correctly.
On Tue, Nov 6, 2012 at 5:00 AM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Regarding the EOFException my guess is that some nodes are acting
> flaky. What version of hadoop are you running?
>
> Brock
>
> On Mon, Nov 5, 2012 at 8:43 PM, Cameron Gandevia <[EMAIL PROTECTED]>
> wrote:
> > Hi
> >
> > I starting noticing the following error on our flume nodes and was
> wondering
> > if anyone had any ideas. I am still trying to figure out if its related
> to
> > something happening in our Hadoop cluster.
> >
> > I am running about 20 of the following sink configurations
> >
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.type = hdfs
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.hdfs.path > > hdfs://${HADOOP_NAMENODE}:8020/rawLogs/%Y-%m-%d/%H00
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.hdfs.codeC > > com.hadoop.compression.lzo.LzopCodec
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.hdfs.fileType = CompressedStream
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.hdfs.rollInterval = 300
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.hdfs.rollSize = 0
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.hdfs.rollCount = 0
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.hdfs.batchSize = 2000
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.hdfs.callTimeout = 60000
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.hdfs.filePrefix > > ${FLUME_COLLECTOR_ID}_20
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.txnEventMax = 1000
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.serializer > > gr_flume_utils.serializer.JSONEventSerializer$Builder
> > ${FLUME_COLLECTOR_ID}.sinks.hdfs-sink20.channel = hdfs-fileChannel
> >
> >
> > 2012-11-06 02:21:03,098 [hdfs-hdfs-sink11-call-runner-5] WARN
> > org.apache.hadoop.hdfs.DFSClient - Error while syncing
> > java.io.EOFException
> > at java.io.DataInputStream.readShort(DataInputStream.java:298)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:3671)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3594)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2400(DFSClient.java:2792)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)
> > 2012-11-06 02:21:03,098 [hdfs-hdfs-sink11-call-runner-5] WARN
> > org.apache.flume.sink.hdfs.BucketWriter - Caught IOException while
> closing
> > file
> > (hdfs://
> van-mang-perf-hadoop-relay.net:8020/rawLogs/2012-10-28/1700/van-mang-perf-flume-collector2-relay-net_11.1352168380390.lzo.tmp
> ).
> > Exception follows.
> > java.io.EOFException
> > at java.io.DataInputStream.readShort(DataInputStream.java:298)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:3671)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3594)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2400(DFSClient.java:2792)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)
> > 2012-11-06 02:21:03,105 [SinkRunner-PollingRunner-DefaultSinkProcessor]
> WARN
> > org.apache.flume.sink.hdfs.HDFSEventSink - HDFS IO error
> > java.io.IOException: write beyond end of stream
> > at
> >
> com.hadoop.compression.lzo.LzopOutputStream.write(LzopOutputStream.java:127)
> > at java.io.OutputStream.write(OutputStream.java:58)
> > at
> >
> org.apache.flume.sink.hdfs.HDFSCompressedDataStream.append(HDFSCompressedDataStream.java:81)
> > at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:328)
> > at
> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:711)
> > at
> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:708)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

Thanks

Cameron Gandevia
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB