Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Is the checkpoint rebuilder still meant to work from main()?


+
Juhani Connolly 2013-02-26, 08:22
Copy link to this message
-
Re: Is the checkpoint rebuilder still meant to work from main()?
Hi Juhani,

Could you please send the full logs? I don't remember if the main method was updated with the new file channel format and also with the encryption patch. The CheckpointRebuilder works fine if you call from the channel starup (using the use-fast-replay parameter).
Thanks,
Hari

--
Hari Shreedharan
On Tuesday, February 26, 2013 at 12:22 AM, Juhani Connolly wrote:

> Since the checkpoint rebuilding process uses up a fair bit more memory
> than normal operation of flume, I noted that there was a main() in
> CheckpointBuilder.java
>
> For personal convenience I've made a patch to run this from the flume-ng
> script, and can of course share it(do we want this for the general release?)
>
> One thing that came up though is that if I delete the contents of the
> checkpoint directory, the first run will error:
>
> Exception in thread "main" java.io.IOException: File
> /tmp/flume-data/log-1.meta has bad version 1c0d0300
> at
> org.apache.flume.channel.file.LogFileFactory.getSequentialReader(LogFileFactory.java:169)
> at
> org.apache.flume.channel.file.CheckpointRebuilder.rebuild(CheckpointRebuilder.java:68)
> at
> org.apache.flume.channel.file.CheckpointRebuilder.main(CheckpointRebuilder.java:257)
>
> The second time I run it, it will go through fine. Things appear to be
> running ok.
>
> Should this still work or is the main() deprecated? Anyone more familiar
> have an idea as to what is causing the exception the first run through?
> Should I make an issue to share this?
>
>
+
Juhani Connolly 2013-02-27, 06:42
+
Hari Shreedharan 2013-02-27, 07:28
+
Hari Shreedharan 2013-02-27, 07:38
+
Hari Shreedharan 2013-02-27, 07:45
+
Juhani Connolly 2013-02-27, 08:14
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB