Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Is the checkpoint rebuilder still meant to work from main()?


Copy link to this message
-
Re: Is the checkpoint rebuilder still meant to work from main()?
Hari Shreedharan 2013-02-27, 07:45
I attached the patch on FLUME-1929 (with a new name). If you find it ok, could you please commit it?

Thanks
Hari

--
Hari Shreedharan
On Tuesday, February 26, 2013 at 11:38 PM, Hari Shreedharan wrote:

> Juhani,
>
> Please try the patch attached and let me know if it works. You were right, we did not update the checkpoint rebuilder when we changed the data file format.
>
>
> Thanks,
> Hari
>
> --
> Hari Shreedharan
>
>
> On Tuesday, February 26, 2013 at 11:28 PM, Hari Shreedharan wrote:
>
> > Juhani,
> >
> > I think I know what the issue is. I will send a patch in a few minutes. You can give it a try and let me know if it works.
> >
> >
> > Thanks,
> > Hari
> > --
> > Hari Shreedharan
> >
> >
> > On Tuesday, February 26, 2013 at 10:42 PM, Juhani Connolly wrote:
> >
> > > Having attempted to run this on logs whose checkpoint was messed up,
> > > running flume started up the checkpoint rebuild process again.
> > >
> > > So it doesn't really look like it is working. I suspect that it is not
> > > capable of recognizing v3 logs but I may be wrong. I'd like to bring
> > > this up to date, but would appreciate it if someone could give me a
> > > heads up on roughly what it would involve, and which code to have a look at.
> > >
> > > On 02/26/2013 06:08 PM, Juhani Connolly wrote:
> > > > Hi Hari,
> > > >
> > > > Here's the logs. Also adding the patch afterwards. Don't think the
> > > > mailinglist accepts attachments so just going to paste it in after the
> > > > logs(note that I changed the parameter for the checkpoint dir so it
> > > > didn't overlap with conf)
> > > >
> > > > # first run
> > > > $ sudo su cy_flume -c "JAVA_HOME=/usr/local/java ./flume-ng cp-rebuild
> > > > -c /etc/flume/conf -h /tmp/flume-check -l /tmp/flume-data/ -t 5000000"
> > > > Info: Sourcing environment configuration script
> > > > /etc/flume/conf/flume-env.sh (http://flume-env.sh)
> > > > + exec /usr/local/java/bin/java -server
> > > > -XX:OnOutOfMemoryError=/tmp/stop.sh (http://stop.sh) -XX:MaxPermSize=24m
> > > > -XX:PermSize=24m -XX:SurvivorRatio=8 -Xmn96m -Xmx512m -Xms128m
> > > > -Dcom.sun.management.jmxremote
> > > > -Dcom.sun.management.jmxremote.port=12345
> > > > -Dcom.sun.management.jmxremote.ssl=false
> > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > -Djava.rmi.server.hostname=172.28.202.76
> > > > -Dflume.monitoring.type=GANGLIA
> > > > -Dflume.monitoring.hosts=pat-log-om01:8649 -cp
> > > > '/etc/flume/conf:/usr/lib/flume/lib/*' -Djava.library.path=
> > > > org.apache.flume.channel.file.CheckpointRebuilder -h /tmp/flume-check
> > > > -l /tmp/flume-data/ -t 5000000
> > > > Exception in thread "main" java.io.IOException: File
> > > > /tmp/flume-data/log-1.meta has bad version 1c0d0300
> > > > at
> > > > org.apache.flume.channel.file.LogFileFactory.getSequentialReader(LogFileFactory.java:169)
> > > > at
> > > > org.apache.flume.channel.file.CheckpointRebuilder.rebuild(CheckpointRebuilder.java:68)
> > > > at
> > > > org.apache.flume.channel.file.CheckpointRebuilder.main(CheckpointRebuilder.java:257)
> > > >
> > > > # second run
> > > >
> > > > $ sudo su cy_flume -c "JAVA_HOME=/usr/local/java ./flume-ng cp-rebuild
> > > > -c /etc/flume/conf -h /tmp/flume-check -l /tmp/flume-data/ -t 5000000"
> > > > Info: Sourcing environment configuration script
> > > > /etc/flume/conf/flume-env.sh (http://flume-env.sh)
> > > > + exec /usr/local/java/bin/java -server
> > > > -XX:OnOutOfMemoryError=/tmp/stop.sh (http://stop.sh) -XX:MaxPermSize=24m
> > > > -XX:PermSize=24m -XX:SurvivorRatio=8 -Xmn96m -Xmx512m -Xms128m
> > > > -Dcom.sun.management.jmxremote
> > > > -Dcom.sun.management.jmxremote.port=12345
> > > > -Dcom.sun.management.jmxremote.ssl=false
> > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > -Djava.rmi.server.hostname=172.28.202.76
> > > > -Dflume.monitoring.type=GANGLIA
> > > > -Dflume.monitoring.hosts=pat-log-om01:8649 -cp
> > > > '/etc/flume/conf:/usr/lib/flume/lib/*' -Djava.library.path=