Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Is the checkpoint rebuilder still meant to work from main()?


Copy link to this message
-
Is the checkpoint rebuilder still meant to work from main()?
Since the checkpoint rebuilding process uses up a fair bit more memory
than normal operation of flume, I noted that there was a main() in
CheckpointBuilder.java

For personal convenience I've made a patch to run this from the flume-ng
script, and can of course share it(do we want this for the general release?)

One thing that came up though is that if I delete the contents of the
checkpoint directory, the first run will error:

Exception in thread "main" java.io.IOException: File
/tmp/flume-data/log-1.meta has bad version 1c0d0300
     at
org.apache.flume.channel.file.LogFileFactory.getSequentialReader(LogFileFactory.java:169)
     at
org.apache.flume.channel.file.CheckpointRebuilder.rebuild(CheckpointRebuilder.java:68)
     at
org.apache.flume.channel.file.CheckpointRebuilder.main(CheckpointRebuilder.java:257)

The second time I run it, it will go through fine. Things appear to be
running ok.

Should this still work or is the main() deprecated? Anyone more familiar
have an idea as to what is causing the exception the first run through?
Should I make an issue to share this?
+
Hari Shreedharan 2013-02-26, 08:27
+
Juhani Connolly 2013-02-27, 06:42
+
Hari Shreedharan 2013-02-27, 07:28
+
Hari Shreedharan 2013-02-27, 07:38
+
Hari Shreedharan 2013-02-27, 07:45
+
Juhani Connolly 2013-02-27, 08:14
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB