Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Is the checkpoint rebuilder still meant to work from main()?


Copy link to this message
-
Re: Is the checkpoint rebuilder still meant to work from main()?
Juhani,

I think I know what the issue is. I will send a patch in a few minutes. You can give it a try and let me know if it works.
Thanks,
Hari
--
Hari Shreedharan
On Tuesday, February 26, 2013 at 10:42 PM, Juhani Connolly wrote:

> Having attempted to run this on logs whose checkpoint was messed up,
> running flume started up the checkpoint rebuild process again.
>
> So it doesn't really look like it is working. I suspect that it is not
> capable of recognizing v3 logs but I may be wrong. I'd like to bring
> this up to date, but would appreciate it if someone could give me a
> heads up on roughly what it would involve, and which code to have a look at.
>
> On 02/26/2013 06:08 PM, Juhani Connolly wrote:
> > Hi Hari,
> >
> > Here's the logs. Also adding the patch afterwards. Don't think the
> > mailinglist accepts attachments so just going to paste it in after the
> > logs(note that I changed the parameter for the checkpoint dir so it
> > didn't overlap with conf)
> >
> > # first run
> > $ sudo su cy_flume -c "JAVA_HOME=/usr/local/java ./flume-ng cp-rebuild
> > -c /etc/flume/conf -h /tmp/flume-check -l /tmp/flume-data/ -t 5000000"
> > Info: Sourcing environment configuration script
> > /etc/flume/conf/flume-env.sh (http://flume-env.sh)
> > + exec /usr/local/java/bin/java -server
> > -XX:OnOutOfMemoryError=/tmp/stop.sh (http://stop.sh) -XX:MaxPermSize=24m
> > -XX:PermSize=24m -XX:SurvivorRatio=8 -Xmn96m -Xmx512m -Xms128m
> > -Dcom.sun.management.jmxremote
> > -Dcom.sun.management.jmxremote.port=12345
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Djava.rmi.server.hostname=172.28.202.76
> > -Dflume.monitoring.type=GANGLIA
> > -Dflume.monitoring.hosts=pat-log-om01:8649 -cp
> > '/etc/flume/conf:/usr/lib/flume/lib/*' -Djava.library.path=
> > org.apache.flume.channel.file.CheckpointRebuilder -h /tmp/flume-check
> > -l /tmp/flume-data/ -t 5000000
> > Exception in thread "main" java.io.IOException: File
> > /tmp/flume-data/log-1.meta has bad version 1c0d0300
> > at
> > org.apache.flume.channel.file.LogFileFactory.getSequentialReader(LogFileFactory.java:169)
> > at
> > org.apache.flume.channel.file.CheckpointRebuilder.rebuild(CheckpointRebuilder.java:68)
> > at
> > org.apache.flume.channel.file.CheckpointRebuilder.main(CheckpointRebuilder.java:257)
> >
> > # second run
> >
> > $ sudo su cy_flume -c "JAVA_HOME=/usr/local/java ./flume-ng cp-rebuild
> > -c /etc/flume/conf -h /tmp/flume-check -l /tmp/flume-data/ -t 5000000"
> > Info: Sourcing environment configuration script
> > /etc/flume/conf/flume-env.sh (http://flume-env.sh)
> > + exec /usr/local/java/bin/java -server
> > -XX:OnOutOfMemoryError=/tmp/stop.sh (http://stop.sh) -XX:MaxPermSize=24m
> > -XX:PermSize=24m -XX:SurvivorRatio=8 -Xmn96m -Xmx512m -Xms128m
> > -Dcom.sun.management.jmxremote
> > -Dcom.sun.management.jmxremote.port=12345
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Djava.rmi.server.hostname=172.28.202.76
> > -Dflume.monitoring.type=GANGLIA
> > -Dflume.monitoring.hosts=pat-log-om01:8649 -cp
> > '/etc/flume/conf:/usr/lib/flume/lib/*' -Djava.library.path=
> > org.apache.flume.channel.file.CheckpointRebuilder -h /tmp/flume-check
> > -l /tmp/flume-data/ -t 5000000
> > $
> >
> >
> > diff from here:
> >
> > diff --git a/bin/flume-ng b/bin/flume-ng
> > index ee86c95..b7174b6 100755
> > --- a/bin/flume-ng
> > +++ b/bin/flume-ng
> > @@ -26,6 +26,7 @@
> > FLUME_AGENT_CLASS="org.apache.flume.node.Application"
> > FLUME_AVRO_CLIENT_CLASS="org.apache.flume.client.avro.AvroCLIClient"
> > FLUME_VERSION_CLASS="org.apache.flume.tools.VersionInfo"
> > +FLUME_CHECKPOINT_REBUILDER_CLASS="org.apache.flume.channel.file.CheckpointRebuilder"
> >
> >
> > CLEAN_FLAG=1
> > ################################
> > @@ -198,6 +199,11 @@ avro-client options:
> > --headerFile,-R <file> File containing event headers as key/value
> > pairs on each new line