Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume error in FIleChannel


Copy link to this message
-
Re: Flume error in FIleChannel
Looks like the file may have been corrupted. Can you verify if you are out
of disk space or can see something that might have caused the data to be
corrupted?

Hari
On Thu, Jun 27, 2013 at 6:41 AM, Rahul Ravindran <[EMAIL PROTECTED]> wrote:

> Hi,
>   We are using CDH flume 1.3 (which ships with 4.2.1). We see this error
> in our flume logs in our production system and restarting flume did not
> help. Looking at the flume code, it appears to be expecting the byte to be
> an OPERATION, but is not. Any ideas on what happened?
>
> Thanks,
> ~Rahul.
>
>
> 27 Jun 2013 05:58:12,246 INFO  [Log-BackgroundWorker-ch3]
> (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)
>  - Updating log-780.meta currentPosition = 68617970, logWriteOrderID > 1400053754907
> 27 Jun 2013 05:58:12,248 INFO  [Log-BackgroundWorker-ch3]
> (org.apache.flume.channel.file.Log.writeCheckpoint:898)  - Updated
> checkpoint for file: /flume3/data/log-780 position: 68617970
> logWriteOrderID: 1400053754907
> 27 Jun 2013 05:58:12,529 INFO  [Log-BackgroundWorker-ch2]
> (org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:108)
>  - Start checkpoint for /flume2/checkpoint/checkpoint, elements to sync > 43941
> 27 Jun 2013 05:58:12,531 INFO  [Log-BackgroundWorker-ch2]
> (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:120)
>  - Updating checkpoint metadata: logWriteOrderID: 1400053760540, queueSize:
> 0, queueHead: 16264802
> 27 Jun 2013 05:58:12,583 INFO  [Log-BackgroundWorker-ch2]
> (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)
>  - Updating log-786.meta currentPosition = 66046989, logWriteOrderID > 1400053760540
> 27 Jun 2013 05:58:12,585 INFO  [Log-BackgroundWorker-ch2]
> (org.apache.flume.channel.file.Log.writeCheckpoint:898)  - Updated
> checkpoint for file: /flume2/data/log-786 position: 66046989
> logWriteOrderID: 1400053760540
> 27 Jun 2013 05:58:17,679 INFO  [Log-BackgroundWorker-ch1]
> (org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:108)
>  - Start checkpoint for /flume1/checkpoint/checkpoint, elements to sync > 225955
> 27 Jun 2013 05:58:17,682 INFO  [Log-BackgroundWorker-ch1]
> (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:120)
>  - Updating checkpoint metadata: logWriteOrderID: 1400053832535, queueSize:
> 7255426, queueHead: 1778328
> 27 Jun 2013 05:58:17,736 INFO  [Log-BackgroundWorker-ch1]
> (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)
>  - Updating log-781.meta currentPosition = 652840345, logWriteOrderID > 1400053832535
> 27 Jun 2013 05:58:17,738 INFO  [Log-BackgroundWorker-ch1]
> (org.apache.flume.channel.file.Log.writeCheckpoint:898)  - Updated
> checkpoint for file: /flume1/data/log-781 position: 652840345
> logWriteOrderID: 1400053832535
> 27 Jun 2013 05:58:17,739 INFO  [Log-BackgroundWorker-ch1]
> (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing
> RandomReader /flume1/data/log-779
> 27 Jun 2013 05:58:17,745 INFO  [Log-BackgroundWorker-ch1]
> (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)
>  - Updating log-779.meta currentPosition = 1599537606, logWriteOrderID > 1400053832535
> 27 Jun 2013 05:58:17,746 INFO  [Log-BackgroundWorker-ch1]
> (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing
> RandomReader /flume1/data/log-780
> 27 Jun 2013 05:58:17,752 INFO  [Log-BackgroundWorker-ch1]
> (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)
>  - Updating log-780.meta currentPosition = 1610002802, logWriteOrderID > 1400053832535
> 27 Jun 2013 05:58:25,538 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.HDFSEventSink.process:457)  - process failed
> java.lang.IllegalStateException: 1
>         at
> com.google.common.base.Preconditions.checkState(Preconditions.java:145)
>         at
> org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:335)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB