Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Log42 embedded Flume agent fails to rollback (log4j2-278)


+
Edward Sargisson 2013-06-10, 22:31
+
Hari Shreedharan 2013-06-10, 22:55
+
Ralph Goers 2013-06-11, 07:40
Copy link to this message
-
Re: Log42 embedded Flume agent fails to rollback (log4j2-278)
Edward Sargisson 2013-06-11, 16:39
Hi Ralph,
If you are using the Log4j2 Flume Appender you are probably using SLF4J -
and there can be only one implementation in use at a time. I would consider
using SLF4J to be best practice unless one is using the nicer interface of
Log4j2.

I do see errors about recursive calls to the appender - but I also get this
stacktrace a lot.

Unless we have some kind of easily insertable StatusLog that Flume can use
(i.e. Hari's suggestion) then I currently agree that Flume and Avro loggers
should be configured to not go via the FlumeAppender. I have yet to test
this configuration but I suspect that the best way is to direct those to
their own special rolling file which rolls every minute into a folder that
Flume reads with the spooling directory source. I have this method working
reasonably well in my development environment but note that it requires
log4j2 because of the concurrency fix.

If this is what we expect people to do then we should document it.

Cheers,
Edward
On Tue, Jun 11, 2013 at 12:40 AM, Ralph Goers <[EMAIL PROTECTED]>wrote:

> I am a bit confused.  At line 566 Log calls SLF4J so you will only end up
> back in Log4j 2 if you have made Log4j 2 the SLF4J implementation. That
> shouldn't cause any problems. Having Flume (or any other code such as
> Hibernate) attempt to log from inside an Appender is expected and would be
> detected and the logging call would be ignored if it is made on the same
> thread.  However, if the work is being done on a second thread it would not
> appear to be a recursive call.
>
> The simple solution for this is to configure a Logger for org.apache.flume
> and do not allow it to be routed to the FlumeAppender.
>
> Ralph
>
>
>
> On Jun 10, 2013, at 3:31 PM, Edward Sargisson wrote:
>
> Hi all,
> I have discovered a defect in the way the Log4j2 embedded Flume agent
> works. As it is an interaction between the two code bases I am taking the
> liberty of making both teams aware of it by cross-posting to both dev lists.
>
> The summary is that, during a rollback, Flume's
> org.apache.flume.channel.file.Log calls the logging subsystem at line 566.
> This recursive call causes problems. There are other places where logging
> is called which cause similar problems.
>
> Details: https://issues.apache.org/jira/browse/LOG4J2-278
>
> Cheers,
> Edward
>
>
>
>