Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: Improvement for Log4j configuration


Copy link to this message
-
Re: Review Request: Improvement for Log4j configuration
Denny Ye 2012-08-03, 02:41


> On Aug. 2, 2012, 10:03 a.m., Mike Percy wrote:
> > The INFO Hadoop setting is certainly needed. Also, including the thread name is great. However I have heard that DailyRollingFileAppender is unreliable and can cause data loss at roll time. What's the reason behind that change?

Thanks Mike, I'm familiar with DatedFileAppender in my project and DailyRollingFileAppender in Hadoop. I think that's good in troubleshooting for me and QA to track issue at specified date. Log name suffixed with 'yyyy-MM-dd' is better than 'flume-{number}.log'. Another tip in my experience is we have tools to clean up log file automatically with expire, it verify file suffix.

Also, following your concern about data loss in DailyRollingFileAppender. I found a Hadoop bug https://issues.apache.org/jira/browse/HADOOP-8149, the same problem and they changed DailyRollingFileAppender back to RollingFileAppender. I don't agree with them completely. It's hard to distinguish log at specified date.

May be the DatedFileAppender is another choose. Do you think so?  
- Denny
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6316/#review9745
-----------------------------------------------------------
On Aug. 2, 2012, 7 a.m., Denny Ye wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6316/
> -----------------------------------------------------------
>
> (Updated Aug. 2, 2012, 7 a.m.)
>
>
> Review request for Flume and Hari Shreedharan.
>
>
> Description
> -------
>
> Updated log4j.properties
> 1. Add 'logs' folder. It's better to aggregate logs into unified folder.
> 2. Log file using DailyRollingFileAppender
> 3. More useful log pattern. Add thread and code line information
> 4. Set debug level for Hadoop. If we use debug level, Hadoop log takes too many lines.
>
>
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1418.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1418
>
>
> Diffs
> -----
>
>   trunk/conf/log4j.properties 1363210
>
> Diff: https://reviews.apache.org/r/6316/diff/
>
>
> Testing
> -------
>
> That's OK in my environment
>
>
> Thanks,
>
> Denny Ye
>
>