Hi,

Could you please point me how to stream the attempt logs that hadoop
generates to chukwa, while at the same time saving them the way hadoop saves
by default under attempt directories/stdout ?

Thanks a lot

--
View this message in context: http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3665088.html
Sent from the Chukwa - Users mailing list archive at Nabble.com.

This can be setup by using TaskLogAppender that comes with Chukwa.
In $HADOOP_CONF_DIR/log4j.properties, modify the TaskLogAppender configuration
to:

log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.TaskLogAppender

Make sure chukwa-0.x.0-client.jar is in Hadoop class path, and restart Hadoop.

This should enable Task Attempt Logs to be sent through Chukwa data
collection system, and also log a copy locally on the task tracker
node.  Hope this is what you are looking for.

regards,
Eric

On Mon, Jan 16, 2012 at 7:20 PM, Abhijit Dhar <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Could you please point me how to stream the attempt logs that hadoop
> generates to chukwa, while at the same time saving them the way hadoop saves
> by default under attempt directories/stdout ?
>
> Thanks a lot
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3665088.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.

I added this in my $HADOOP_CONF_DIR/log4j.properties

log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.TaskLogAppender
log4j.appender.TLA.recordType=CrawlerLog
log4j.appender.TLA.taskId=${hadoop.tasklog.taskid}
log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t]
[%c{5}.%M()]:: %m%n

But, it looks like when I add this to the rootlogger
(log4j.rootLogger=ERROR, stdout, TLA), it stops going through even my
stdout log appender.
Any idea, what might be going on? (chukwa jar is on my classpath, and I
don't see any ClassNotFound exception, so I think that part is good)

Thanks,
On Tue, Jan 17, 2012 at 6:03 PM, Eric Yang <[EMAIL PROTECTED]> wrote:

> This can be setup by using TaskLogAppender that comes with Chukwa.
> In $HADOOP_CONF_DIR/log4j.properties, modify the TaskLogAppender
> configuration
> to:
>
>
> log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.TaskLogAppender
>
> Make sure chukwa-0.x.0-client.jar is in Hadoop class path, and restart
> Hadoop.
>
> This should enable Task Attempt Logs to be sent through Chukwa data
> collection system, and also log a copy locally on the task tracker
> node.  Hope this is what you are looking for.
>
> regards,
> Eric
>
> On Mon, Jan 16, 2012 at 7:20 PM, Abhijit Dhar <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > Could you please point me how to stream the attempt logs that hadoop
> > generates to chukwa, while at the same time saving them the way hadoop
> saves
> > by default under attempt directories/stdout ?
> >
> > Thanks a lot
> >
> > --
> > View this message in context:
> http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3665088.html
> > Sent from the Chukwa - Users mailing list archive at Nabble.com.
>

--
Абижит Дар

The logger should set to INFO because the information are logged at
INFO level.  The increased severity level to ERROR filtered out all
log entries from task attempts.
Hope this helps.

regards,
Eric
On Thu, Jan 19, 2012 at 6:13 PM, Abhijit Dhar <[EMAIL PROTECTED]> wrote:
> I added this in my $HADOOP_CONF_DIR/log4j.properties
>
> log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.TaskLogAppender
> log4j.appender.TLA.recordType=CrawlerLog
> log4j.appender.TLA.taskId=${hadoop.tasklog.taskid}
> log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
> log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t]
> [%c{5}.%M()]:: %m%n
>
> But, it looks like when I add this to the rootlogger
> (log4j.rootLogger=ERROR, stdout, TLA), it stops going through even my stdout
> log appender.
> Any idea, what might be going on? (chukwa jar is on my classpath, and I
> don't see any ClassNotFound exception, so I think that part is good)
>
> Thanks,
>
>
>
> On Tue, Jan 17, 2012 at 6:03 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>
>> This can be setup by using TaskLogAppender that comes with Chukwa.
>> In $HADOOP_CONF_DIR/log4j.properties, modify the TaskLogAppender
>> configuration
>> to:
>>
>>
>> log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.TaskLogAppender
>>
>> Make sure chukwa-0.x.0-client.jar is in Hadoop class path, and restart
>> Hadoop.
>>
>> This should enable Task Attempt Logs to be sent through Chukwa data
>> collection system, and also log a copy locally on the task tracker
>> node.  Hope this is what you are looking for.
>>
>> regards,
>> Eric
>>
>> On Mon, Jan 16, 2012 at 7:20 PM, Abhijit Dhar <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> >
>> > Could you please point me how to stream the attempt logs that hadoop
>> > generates to chukwa, while at the same time saving them the way hadoop
>> > saves
>> > by default under attempt directories/stdout ?
>> >
>> > Thanks a lot
>> >
>> > --
>> > View this message in context:
>> > http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3665088.html
>> > Sent from the Chukwa - Users mailing list archive at Nabble.com.
>
>
>
>
> --
> Абижит Дар

I tried different loglevels too. But, I checked, and found that
TaskLogAppender does not even register itself with the chuka agent as an
adapter. I tried using ChukwaDailyRollingFileAppender with the below config,
and it works flawlessly -

log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
log4j.appender.TLA.File=logs/jcrawler.log
log4j.appender.TLA.recordType=CrawlerLog
log4j.appender.TLA.chukwaClientHostname=localhost
log4j.appender.TLA.chukwaClientPortNum=9093
log4j.appender.TLA.DatePattern=.yyyy-MM-dd
log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t]
[%c{5}.%M()]:: %m%n

But, with ChukwaDailyRollingFileAppender, it does not create the task log
attempt directories(which I need too)

--
View this message in context: http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3674393.html
Sent from the Chukwa - Users mailing list archive at Nabble.com.

Chukwa's own TaskLogAppender code seems to be using default
initialization parameters for contacting Chukwa Agent.  The code may
need some update for it to work right.  The subAppend function between
ChukwaDailyRollingFileAppender and TaskLogAppender should look very
similar.  Unfortunately, it isn't at the current state.  I think a
patch is required to get Chukwa's TaskLogAppender to work.

regards,
Eric

On Jan 19, 2012, at 6:49 PM, Abhijit Dhar <[EMAIL PROTECTED]> wrote:

> I tried different loglevels too. But, I checked, and found that
> TaskLogAppender does not even register itself with the chuka agent as an
> adapter. I tried using ChukwaDailyRollingFileAppender with the below config,
> and it works flawlessly -
>
> log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
> log4j.appender.TLA.File=logs/jcrawler.log
> log4j.appender.TLA.recordType=CrawlerLog
> log4j.appender.TLA.chukwaClientHostname=localhost
> log4j.appender.TLA.chukwaClientPortNum=9093
> log4j.appender.TLA.DatePattern=.yyyy-MM-dd
> log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
> log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t]
> [%c{5}.%M()]:: %m%n
>
> But, with ChukwaDailyRollingFileAppender, it does not create the task log
> attempt directories(which I need too)
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3674393.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.

Can you point me how to fix this thing? Or if there is a workaround?
Getting this to work would be really helpful.

On Thu, Jan 19, 2012 at 8:59 PM, Eric Yang <[EMAIL PROTECTED]> wrote:

> Chukwa's own TaskLogAppender code seems to be using default
> initialization parameters for contacting Chukwa Agent.  The code may
> need some update for it to work right.  The subAppend function between
> ChukwaDailyRollingFileAppender and TaskLogAppender should look very
> similar.  Unfortunately, it isn't at the current state.  I think a
> patch is required to get Chukwa's TaskLogAppender to work.
>
> regards,
> Eric
>
> On Jan 19, 2012, at 6:49 PM, Abhijit Dhar <[EMAIL PROTECTED]>
> wrote:
>
> > I tried different loglevels too. But, I checked, and found that
> > TaskLogAppender does not even register itself with the chuka agent as an
> > adapter. I tried using ChukwaDailyRollingFileAppender with the below
> config,
> > and it works flawlessly -
> >
> >
> log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
> > log4j.appender.TLA.File=logs/jcrawler.log
> > log4j.appender.TLA.recordType=CrawlerLog
> > log4j.appender.TLA.chukwaClientHostname=localhost
> > log4j.appender.TLA.chukwaClientPortNum=9093
> > log4j.appender.TLA.DatePattern=.yyyy-MM-dd
> > log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
> > log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t]
> > [%c{5}.%M()]:: %m%n
> >
> > But, with ChukwaDailyRollingFileAppender, it does not create the task log
> > attempt directories(which I need too)
> >
> > --
> > View this message in context:
> http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3674393.html
> > Sent from the Chukwa - Users mailing list archive at Nabble.com.
>

--
Абижит Дар

You can try to merge the code from ChukwaDailyRollingFileAppender with
TaskLogAppender to get a working TaskLogAppender.  In particular, the
subAppend function should be identical to
ChukwaDailyRollingFileAppender.  This should be a jira for Chukwa 0.6.
Hope this helps.

regards,
Eric

On Fri, Jan 20, 2012 at 8:16 PM, Abhijit Dhar <[EMAIL PROTECTED]> wrote:
> Can you point me how to fix this thing? Or if there is a workaround? Getting
> this to work would be really helpful.
>
>
>
>
> On Thu, Jan 19, 2012 at 8:59 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>
>> Chukwa's own TaskLogAppender code seems to be using default
>> initialization parameters for contacting Chukwa Agent.  The code may
>> need some update for it to work right.  The subAppend function between
>> ChukwaDailyRollingFileAppender and TaskLogAppender should look very
>> similar.  Unfortunately, it isn't at the current state.  I think a
>> patch is required to get Chukwa's TaskLogAppender to work.
>>
>> regards,
>> Eric
>>
>> On Jan 19, 2012, at 6:49 PM, Abhijit Dhar <[EMAIL PROTECTED]>
>> wrote:
>>
>> > I tried different loglevels too. But, I checked, and found that
>> > TaskLogAppender does not even register itself with the chuka agent as an
>> > adapter. I tried using ChukwaDailyRollingFileAppender with the below
>> > config,
>> > and it works flawlessly -
>> >
>> >
>> > log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
>> > log4j.appender.TLA.File=logs/jcrawler.log
>> > log4j.appender.TLA.recordType=CrawlerLog
>> > log4j.appender.TLA.chukwaClientHostname=localhost
>> > log4j.appender.TLA.chukwaClientPortNum=9093
>> > log4j.appender.TLA.DatePattern=.yyyy-MM-dd
>> > log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
>> > log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t]
>> > [%c{5}.%M()]:: %m%n
>> >
>> > But, with ChukwaDailyRollingFileAppender, it does not create the task
>> > log
>> > attempt directories(which I need too)
>> >
>> > --
>> > View this message in context:
>> > http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3674393.html
>> > Sent from the Chukwa - Users mailing list archive at Nabble.com.
>
>
>
>
> --
> Абижит Дар