Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request: Improvement for Log4j configuration


Copy link to this message
-
Re: Review Request: Improvement for Log4j configuration


> On Aug. 2, 2012, 10:03 a.m., Mike Percy wrote:
> > The INFO Hadoop setting is certainly needed. Also, including the thread name is great. However I have heard that DailyRollingFileAppender is unreliable and can cause data loss at roll time. What's the reason behind that change?
>
> Denny Ye wrote:
>     Thanks Mike, I'm familiar with DatedFileAppender in my project and DailyRollingFileAppender in Hadoop. I think that's good in troubleshooting for me and QA to track issue at specified date. Log name suffixed with 'yyyy-MM-dd' is better than 'flume-{number}.log'. Another tip in my experience is we have tools to clean up log file automatically with expire, it verify file suffix.
>    
>     Also, following your concern about data loss in DailyRollingFileAppender. I found a Hadoop bug https://issues.apache.org/jira/browse/HADOOP-8149, the same problem and they changed DailyRollingFileAppender back to RollingFileAppender. I don't agree with them completely. It's hard to distinguish log at specified date.
>    
>     May be the DatedFileAppender is another choose. Do you think so?
>
> Mike Percy wrote:
>     I've never used DatedFileAppender personally. I believe this is the original site: http://minaret.biz/tips/datedFileAppender.html and someone apparently made some modifications to it: http://sourceforge.net/p/log4j-dfa/home/Home/
>    
>     Do you know of anyone using it in production? If not, it seems a bit hasty to just make it the default for all users of Flume...

We used DatedFileAppender in production two years. It's stable always.
Configuration example in log4j.xml in my project:
<param name="Threshold" value="DEBUG" />
<param name="Directory" value="${catalina.base}/logs" />
<param name="Prefix" value="xxxxx." />
<param name="Suffix" value=".log" />
<param name="Append" value="true" />
<layout class="org.apache.log4j.PatternLayout">
     <param name="ConversionPattern" value="%d{dd MMM yyyy HH:mm:ss,SSS} [%t] %-5p (%C.%M:%L) %x - %m%n" />
</layout>

Log file will be : xxxxx.2012-08-03.log
- Denny
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6316/#review9745
-----------------------------------------------------------
On Aug. 2, 2012, 7 a.m., Denny Ye wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6316/
> -----------------------------------------------------------
>
> (Updated Aug. 2, 2012, 7 a.m.)
>
>
> Review request for Flume and Hari Shreedharan.
>
>
> Description
> -------
>
> Updated log4j.properties
> 1. Add 'logs' folder. It's better to aggregate logs into unified folder.
> 2. Log file using DailyRollingFileAppender
> 3. More useful log pattern. Add thread and code line information
> 4. Set debug level for Hadoop. If we use debug level, Hadoop log takes too many lines.
>
>
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1418.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1418
>
>
> Diffs
> -----
>
>   trunk/conf/log4j.properties 1363210
>
> Diff: https://reviews.apache.org/r/6316/diff/
>
>
> Testing
> -------
>
> That's OK in my environment
>
>
> Thanks,
>
> Denny Ye
>
>