Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Is the checkpoint rebuilder still meant to work from main()?


Copy link to this message
-
Re: Is the checkpoint rebuilder still meant to work from main()?
Juhani Connolly 2013-02-27, 06:42
Having attempted to run this on logs whose checkpoint was messed up,
running flume started up the checkpoint rebuild process again.

So it doesn't really look like it is working. I suspect that it is not
capable of recognizing v3 logs but I may be wrong. I'd like to bring
this up to date, but would appreciate it if someone could give me a
heads up on roughly what it would involve, and which code to have a look at.

On 02/26/2013 06:08 PM, Juhani Connolly wrote:
> Hi Hari,
>
> Here's the logs. Also adding the patch afterwards. Don't think the
> mailinglist accepts attachments so just going to paste it in after the
> logs(note that I changed the parameter for the checkpoint dir so it
> didn't overlap with conf)
>
> # first run
> $ sudo su cy_flume -c "JAVA_HOME=/usr/local/java ./flume-ng cp-rebuild
> -c /etc/flume/conf -h /tmp/flume-check -l /tmp/flume-data/ -t 5000000"
> Info: Sourcing environment configuration script
> /etc/flume/conf/flume-env.sh
> + exec /usr/local/java/bin/java -server
> -XX:OnOutOfMemoryError=/tmp/stop.sh -XX:MaxPermSize=24m
> -XX:PermSize=24m -XX:SurvivorRatio=8 -Xmn96m -Xmx512m -Xms128m
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=12345
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Djava.rmi.server.hostname=172.28.202.76
> -Dflume.monitoring.type=GANGLIA
> -Dflume.monitoring.hosts=pat-log-om01:8649 -cp
> '/etc/flume/conf:/usr/lib/flume/lib/*' -Djava.library.path=
> org.apache.flume.channel.file.CheckpointRebuilder -h /tmp/flume-check
> -l /tmp/flume-data/ -t 5000000
> Exception in thread "main" java.io.IOException: File
> /tmp/flume-data/log-1.meta has bad version 1c0d0300
>     at
> org.apache.flume.channel.file.LogFileFactory.getSequentialReader(LogFileFactory.java:169)
>     at
> org.apache.flume.channel.file.CheckpointRebuilder.rebuild(CheckpointRebuilder.java:68)
>     at
> org.apache.flume.channel.file.CheckpointRebuilder.main(CheckpointRebuilder.java:257)
>
> # second run
>
> $ sudo su cy_flume -c "JAVA_HOME=/usr/local/java ./flume-ng cp-rebuild
> -c /etc/flume/conf -h /tmp/flume-check -l /tmp/flume-data/ -t 5000000"
> Info: Sourcing environment configuration script
> /etc/flume/conf/flume-env.sh
> + exec /usr/local/java/bin/java -server
> -XX:OnOutOfMemoryError=/tmp/stop.sh -XX:MaxPermSize=24m
> -XX:PermSize=24m -XX:SurvivorRatio=8 -Xmn96m -Xmx512m -Xms128m
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=12345
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Djava.rmi.server.hostname=172.28.202.76
> -Dflume.monitoring.type=GANGLIA
> -Dflume.monitoring.hosts=pat-log-om01:8649 -cp
> '/etc/flume/conf:/usr/lib/flume/lib/*' -Djava.library.path=
> org.apache.flume.channel.file.CheckpointRebuilder -h /tmp/flume-check
> -l /tmp/flume-data/ -t 5000000
> $
>
>
> diff from here:
>
> diff --git a/bin/flume-ng b/bin/flume-ng
> index ee86c95..b7174b6 100755
> --- a/bin/flume-ng
> +++ b/bin/flume-ng
> @@ -26,6 +26,7 @@
>  FLUME_AGENT_CLASS="org.apache.flume.node.Application"
>  FLUME_AVRO_CLIENT_CLASS="org.apache.flume.client.avro.AvroCLIClient"
>  FLUME_VERSION_CLASS="org.apache.flume.tools.VersionInfo"
> +FLUME_CHECKPOINT_REBUILDER_CLASS="org.apache.flume.channel.file.CheckpointRebuilder"
>
>
>  CLEAN_FLAG=1
>  ################################
> @@ -198,6 +199,11 @@ avro-client options:
>    --headerFile,-R <file> File containing event headers as key/value
> pairs on each new line
>    --help,-h              display help text
>
> +cp-rebuild options:
> +  -h <dir>               Checkpoint directory
> +  -l <dir>[,<dir>]*      Comma-separated list of log directories
> +  -t <cap>               capacity of the channel
> +
>    Either --rpcProps or both --host and --port must be specified.
>
>  Note that if <conf> directory is specified, then it is always
> included first
> @@ -260,6 +266,9 @@ case "$mode" in
>     opt_version=1