Our team has used chukwa *CharFileTailingAdaptorUTF8* to collect the log4j
rotated log files for several months.It does help us to collect the logs
from everywhere to our hadoop center.
During the work , we met several problems . And i have raised them in this
mail list , but i still haven't got a good solution.
So we  read the source code , and did some changes

Our log files are generated by the log4j ,and the log4j appender is
org.apache.log4j.DailyRollingFileAppender.
If you use log4j to generate the rotated log ,may this mail will help you.

These two problems are the causes why we have to modify the source code.

1. The mismatching checkpoint size and file size.

     I raised this problem in May 14 ,"the check point offset is bigger
than the log file size". And Ariel Rabkin  and Eric have answered my
question , thanks for your replies.

     When chukwa starts, it will read the the check point file , let the
size be the filereadoffset. The size in the checkpoint indicates how many
bytes the adaptor has send .

     If the log source is stream or a file won't rotate , this size is
right ,it indeed is the filereadoffset.But the file is rorated , the
checkpoint size is often bigger than the file size ,and this will cause
chukwa resend all the log file.

     So we add a "log.info("chunk seqID:"+c.getSeqID());" in
ChukwaHttpSender:send.

     *for (Chunk c : toSend) {
      DataOutputBuffer b = new
DataOutputBuffer(c.getSerializedSizeEstimate());
      try {
        c.write(b);
      } catch (IOException err) {
        log.error("serialization threw IOException", err);
      }
      serializedEvents.add(b);
      // store a CLE for this chunk which we will use to ack this chunk to
the
      // caller of send()
      // (e.g. the agent will use the list of CLE's for checkpointing)
      log.info("chunk seqID:"+c.getSeqID());
      commitResults.add(new CommitListEntry(c.getInitiator(), c.getSeqID(),
         c.getSeqID() - c.getData().length));
    }*
     *
    **The seqid is the offset of the send chunks in this log file.**
   * So when we need to restart the chukwa, we just need to stop the chukwa
, change the size in checkpoint to the last chunk seqid in log and start
chukwa.
      We also can directly apply the seqID to checkpoint size ,but we don't
know if this will cause other problems.
*

*2.* *The method tailFile in FileTailingAdaptor is the core code of
collecting the log. The code use the fileReadOffset , file length to detect
the rotated file.
        *RandomAccessFile newReader = new RandomAccessFile(toWatch, "r");*
*        len = reader.length();*
*        long newLength = newReader.length();*
*        if (newLength < len && fileReadOffset >= len) {*
*          if (reader != null) {*
*            reader.close();*
*          }*
*          *
*          reader = newReader;*
*          fileReadOffset = 0L;*
*          log.debug("Adaptor|"+ adaptorID + "| File size mismatched,
rotating: "*
*              + toWatch.getAbsolutePath());*
*        } else {*
*          try {*
*            if (newReader != null) {*
*              newReader.close();*
*            }*
*            newReader =null;*
*          } catch (Throwable e) {*
*            // do nothing.*
*          }*
*        }*
*
*
*     *This arithmetic does work in most cases. But there is a case ,that
when chukwa starts , the log file is 0 and it will be 0 untill it has been
rotated. After it has been rotated ,becase its size is 0 ,this log will be
removed. A new file has generated , and its size isn't 0.
      But the len is still 0 ,newLength is > 0.So this contition  if
(newLength < len && fileReadOffset >= len)  will never be archived. The new
log file will never be detected.

        So we changed the implemention of this method, we use timestamp to
detect the new log file.The lastSlurpTime is the timestamp of the last
slurp ,it is been declared and assigned in LWFTAdaptor .
       try {
                len = reader.length();
                if(lastSlurpTime == 0){
                    lastSlurpTime = System.currentTimeMillis();
                }
                if (offsetOfFirstByte > fileReadOffset) {
                    // If the file rotated, the recorded offsetOfFirstByte
is greater than
                    // file size,reset the first byte position to beginning
of the file.
                    fileReadOffset = 0;
                    offsetOfFirstByte = 0L;
                    log.warn("offsetOfFirstByte>fileReadOffset, resetting
offset to 0");
                }
                if (len == fileReadOffset) {
                    File fixedNameFile = new
File(toWatch.getAbsolutePath());
                    long fixedNameLastModified fixedNameFile.lastModified();
                    if (fixedNameLastModified > lastSlurpTime) {
                        // If len == fileReadOffset,the file stops rolling
log or the file has rotated.
                        // But fixedNameLastModified > lastSlurpTime , this
means after the last slurping,the file has been written .
                        // so the file has been rotated.
                        boolean hasLeftData = true;
                        while(hasLeftData){// read the possiblly generated
log
                            hasLeftData = slurp(len, reader);
                        }
                        RandomAccessFile newReader = new
RandomAccessFile(toWatch, "r");
                        if (reader != null) {
                            reader.close();
                        }
                        reader = newReader;
                        fileReadOffset = 0L;
                        len = reader.length();
                        log.debug("Adaptor|" + adaptorID + "| File size
mismatched, rotating: " +
toWatch.getAbsolutePath());
                    }
                    hasMoreData = slurp(len, reader);
                } else if (len < fileReadOffset) {
                    // file has rotated and no detection
                    if (reader != null) {
        

Ivy,

Thanks alot for your contributions!

However, a better way to submit patches to Chukwa is:

1-Open a JIRA issue on

https://issues.apache.org/jira/browse/CHUKWA

I think you will need to register on JIRA first in order to create issues

2- Generate a .patch file using eclipse or command line tools, the .patch
file will be a single file with all the changes you have done.
3- Using the "Submit Patch" option on the JIRA issue upload the file. Your
changes will be reviewed by a committer and committed if everything i okay.

Thank you again for your interest in Chukwa and we really appreciate your
proactive approach in contributing back to the project!

On Tue, Jul 10, 2012 at 10:04 AM, IvyTang <[EMAIL PROTECTED]> wrote:

> Our team has used chukwa *CharFileTailingAdaptorUTF8* to collect the
> log4j rotated log files for several months.It does help us to collect the
> logs from everywhere to our hadoop center.
> During the work , we met several problems . And i have raised them in this
> mail list , but i still haven't got a good solution.
> So we  read the source code , and did some changes
>
> Our log files are generated by the log4j ,and the log4j appender is
> org.apache.log4j.DailyRollingFileAppender.
> If you use log4j to generate the rotated log ,may this mail will help you.
>
> These two problems are the causes why we have to modify the source code.
>
> 1. The mismatching checkpoint size and file size.
>
>      I raised this problem in May 14 ,"the check point offset is bigger
> than the log file size". And Ariel Rabkin  and Eric have answered my
> question , thanks for your replies.
>
>      When chukwa starts, it will read the the check point file , let the
> size be the filereadoffset. The size in the checkpoint indicates how many
> bytes the adaptor has send .
>
>      If the log source is stream or a file won't rotate , this size is
> right ,it indeed is the filereadoffset.But the file is rorated , the
> checkpoint size is often bigger than the file size ,and this will cause
> chukwa resend all the log file.
>
>      So we add a "log.info("chunk seqID:"+c.getSeqID());" in
> ChukwaHttpSender:send.
>
>      *for (Chunk c : toSend) {
>       DataOutputBuffer b = new
> DataOutputBuffer(c.getSerializedSizeEstimate());
>       try {
>         c.write(b);
>       } catch (IOException err) {
>         log.error("serialization threw IOException", err);
>       }
>       serializedEvents.add(b);
>       // store a CLE for this chunk which we will use to ack this chunk to
> the
>       // caller of send()
>       // (e.g. the agent will use the list of CLE's for checkpointing)
>       log.info("chunk seqID:"+c.getSeqID());
>       commitResults.add(new CommitListEntry(c.getInitiator(),
> c.getSeqID(),
>          c.getSeqID() - c.getData().length));
>     }*
>      *
>     **The seqid is the offset of the send chunks in this log file.**
>    * So when we need to restart the chukwa, we just need to stop the
> chukwa , change the size in checkpoint to the last chunk seqid in log and
> start chukwa.
>       We also can directly apply the seqID to checkpoint size ,but we
> don't know if this will cause other problems.
> *
>
> *2.* *The method tailFile in FileTailingAdaptor is the core code of
> collecting the log. The code use the fileReadOffset , file length to detect
> the rotated file.
>         *RandomAccessFile newReader = new RandomAccessFile(toWatch, "r");*
> *        len = reader.length();*
> *        long newLength = newReader.length();*
> *        if (newLength < len && fileReadOffset >= len) {*
> *          if (reader != null) {*
> *            reader.close();*
> *          }*
> *          *
> *          reader = newReader;*
> *          fileReadOffset = 0L;*
> *          log.debug("Adaptor|"+ adaptorID + "| File size mismatched,
> rotating: "*
> *              + toWatch.getAbsolutePath());*
> *        } else {*
> *          try {*
> *            if (newReader != null) {*
> *              newReader.close();*
> *            }*
Ahmed Fathalla

Thanks for you reply!

 I will do this in JIRA issue.

On Tue, Jul 10, 2012 at 4:36 PM, Ahmed Fathalla <[EMAIL PROTECTED]> wrote:

> Ivy,
>
> Thanks alot for your contributions!
>
> However, a better way to submit patches to Chukwa is:
>
> 1-Open a JIRA issue on
>
> https://issues.apache.org/jira/browse/CHUKWA
>
> I think you will need to register on JIRA first in order to create issues
>
> 2- Generate a .patch file using eclipse or command line tools, the .patch
> file will be a single file with all the changes you have done.
>
>
> 3- Using the "Submit Patch" option on the JIRA issue upload the file. Your
> changes will be reviewed by a committer and committed if everything i okay.
>
> Thank you again for your interest in Chukwa and we really appreciate your
> proactive approach in contributing back to the project!
>
> On Tue, Jul 10, 2012 at 10:04 AM, IvyTang <[EMAIL PROTECTED]> wrote:
>
>> Our team has used chukwa *CharFileTailingAdaptorUTF8* to collect the
>> log4j rotated log files for several months.It does help us to collect the
>> logs from everywhere to our hadoop center.
>> During the work , we met several problems . And i have raised them in
>> this mail list , but i still haven't got a good solution.
>> So we  read the source code , and did some changes
>>
>> Our log files are generated by the log4j ,and the log4j appender is
>> org.apache.log4j.DailyRollingFileAppender.
>> If you use log4j to generate the rotated log ,may this mail will help you.
>>
>> These two problems are the causes why we have to modify the source code.
>>
>> 1. The mismatching checkpoint size and file size.
>>
>>      I raised this problem in May 14 ,"the check point offset is bigger
>> than the log file size". And Ariel Rabkin  and Eric have answered my
>> question , thanks for your replies.
>>
>>      When chukwa starts, it will read the the check point file , let the
>> size be the filereadoffset. The size in the checkpoint indicates how many
>> bytes the adaptor has send .
>>
>>      If the log source is stream or a file won't rotate , this size is
>> right ,it indeed is the filereadoffset.But the file is rorated , the
>> checkpoint size is often bigger than the file size ,and this will cause
>> chukwa resend all the log file.
>>
>>      So we add a "log.info("chunk seqID:"+c.getSeqID());" in
>> ChukwaHttpSender:send.
>>
>>      *for (Chunk c : toSend) {
>>       DataOutputBuffer b = new
>> DataOutputBuffer(c.getSerializedSizeEstimate());
>>       try {
>>         c.write(b);
>>       } catch (IOException err) {
>>         log.error("serialization threw IOException", err);
>>       }
>>       serializedEvents.add(b);
>>       // store a CLE for this chunk which we will use to ack this chunk
>> to the
>>       // caller of send()
>>       // (e.g. the agent will use the list of CLE's for checkpointing)
>>       log.info("chunk seqID:"+c.getSeqID());
>>       commitResults.add(new CommitListEntry(c.getInitiator(),
>> c.getSeqID(),
>>          c.getSeqID() - c.getData().length));
>>     }*
>>      *
>>     **The seqid is the offset of the send chunks in this log file.**
>>    * So when we need to restart the chukwa, we just need to stop the
>> chukwa , change the size in checkpoint to the last chunk seqid in log and
>> start chukwa.
>>       We also can directly apply the seqID to checkpoint size ,but we
>> don't know if this will cause other problems.
>> *
>>
>> *2.* *The method tailFile in FileTailingAdaptor is the core code of
>> collecting the log. The code use the fileReadOffset , file length to detect
>> the rotated file.
>>         *RandomAccessFile newReader = new RandomAccessFile(toWatch, "r");
>> *
>> *        len = reader.length();*
>> *        long newLength = newReader.length();*
>> *        if (newLength < len && fileReadOffset >= len) {*
>> *          if (reader != null) {*
>> *            reader.close();*
>> *          }*
>> *          *
>> *          reader = newReader;*
>> *          fileReadOffset = 0L;*
>> *          log.debug("Adaptor|"+ adaptorID + "| File size mismatched,

Best regards,

Ivy Tang