Chukwa, mail # dev - Need help to configure chukwa to write logs file generated by log4j to hdfs

Need help to configure chukwa to write logs file generated by log4j to hdfs

NehaS Singh 2012-12-26, 06:19

Hi,
I am using chukwa0.5.I have followed the steps in http://wiki.apache.org/hadoop/Chukwa_Quick_Start site but still I cannot see logs files in hdfs.
Secondly demuxprocessing is blank .Also I m not able to write into hbase.
Please help

Regards,
Neha Singh
________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"

Re: Need help to configure chukwa to write logs file generated by log4j to hdfs

Eric Yang 2012-12-26, 18:06

Hi Neha,

By default, the pipeline writer does not have seqFileWriter enabled, hence,
it does not write to Hadoop. In addition, HADOOP_CONF_DIR and
HBASE_CONF_DIR should be configured in the classpath of the collector.
This can be configured in chukwa-env.sh.
If this still doesn't work, check collector log file for actual error.

Hope this helps.

regards,
Eric

On Tue, Dec 25, 2012 at 10:19 PM, NehaS Singh
<[EMAIL PROTECTED]>wrote:

>
> Hi,
> I am using chukwa0.5.I have followed the steps in
> http://wiki.apache.org/hadoop/Chukwa_Quick_Start site but still I cannot
> see logs files in hdfs.
> Secondly demuxprocessing is blank .Also I m not able to write into hbase.
> Please help
>
> Regards,
> Neha Singh
>
>
> ________________________________
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>

Re: Need help to configure chukwa to write logs file generated by log4j to hdfs

Neha Singh 2012-12-27, 06:58

I have added the HBASE_CONF_DIR as well as HADOOP_CONF_DIR to the classpath
and my collector.og file looks as below
2012-12-20 16:43:43,919 INFO main ChukwaConfiguration - chukwaConf is
/home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/etc/chukwa
2012-12-20 16:43:44,244 INFO main log - Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2012-12-20 16:43:44,288 INFO main log - jetty-6.1.26
2012-12-20 16:43:44,331 INFO main root - initing servletCollector
2012-12-20 16:43:44,334 INFO main SeqFileWriter - rotateInterval is 300000
2012-12-20 16:43:44,334 INFO main SeqFileWriter - not using fixed time
interval scheme
2012-12-20 16:43:44,334 INFO main SeqFileWriter - outputDir is
/home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs
2012-12-20 16:43:44,334 INFO main SeqFileWriter - fsname is
hdfs://172.20.104.226:8020
2012-12-20 16:43:44,334 INFO main SeqFileWriter - filesystem type from
core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
2012-12-20 16:43:44,816 INFO main log - Started
SelectChannelConnector@0.0.0.0:9080
2012-12-20 16:43:45,331 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:43:45,805 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:44:15,805 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:44:45,331 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:44:45,805 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:45:15,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:45:45,331 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:45:45,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:46:15,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:46:45,331 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:46:45,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:47:15,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:47:45,331 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:47:45,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:48:15,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:48:44,805 INFO Timer-2 SeqFileWriter - closing sink
file/home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs/20121220164344752_INFPH01464U_1de46d0$
2012-12-20 16:48:44,855 INFO Timer-2 SeqFileWriter - no chunks written to
/home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs/20121220164344752_INFPH01464U_1de$
2012-12-20 16:48:45,332 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:48:45,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:49:15,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:49:45,332 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:49:45,807 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:50:15,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:50:45,332 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:50:45,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:51:15,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:51:45,332 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:51:45,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:52:15,806 INFO Timer-3 SeqFileWriter -
stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:52:45,332 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0

I am not able to understand the error.could u please help.

View this message in context: http://apache-chukwa.679492.n3.nabble.com/Need-help-to-configure-chukwa-to-write-logs-file-generated-by-log4j-to-hdfs-tp4025511p4025515.html
Sent from the Chukwa - Developers mailing list archive at Nabble.com.

Re: Need help to configure chukwa to write logs file generated by log4j to hdfs

Eric Yang 2012-12-27, 09:06

There doesn't seem to be any errors, nor data received by the collectors.
Check for errors in agent log file, and make sure there are adaptors
configured to collect data.

regards,
Eric

On Wed, Dec 26, 2012 at 10:58 PM, Neha Singh <[EMAIL PROTECTED]>wrote:

> I have added the HBASE_CONF_DIR as well as HADOOP_CONF_DIR to the classpath
> and my collector.og file looks as below
>
>
> 2012-12-20 16:43:43,919 INFO main ChukwaConfiguration - chukwaConf is
> /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/etc/chukwa
> 2012-12-20 16:43:44,244 INFO main log - Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2012-12-20 16:43:44,288 INFO main log - jetty-6.1.26
> 2012-12-20 16:43:44,331 INFO main root - initing servletCollector
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - rotateInterval is 300000
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - not using fixed time
> interval scheme
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - outputDir is
> /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - fsname is
> hdfs://172.20.104.226:8020
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2012-12-20 16:43:44,816 INFO main log - Started
> SelectChannelConnector@0.0.0.0:9080
> 2012-12-20 16:43:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:43:45,805 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:44:15,805 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:44:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:44:45,805 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:45:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:45:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:45:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:46:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:46:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:46:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:47:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:47:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:47:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:48:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:48:44,805 INFO Timer-2 SeqFileWriter - closing sink
>
> file/home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs/20121220164344752_INFPH01464U_1de46d0$
> 2012-12-20 16:48:44,855 INFO Timer-2 SeqFileWriter - no chunks written to
>
> /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs/20121220164344752_INFPH01464U_1de$
> 2012-12-20 16:48:45,332 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:48:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:49:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:49:45,332 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:49:45,807 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:50:15,806 INFO Timer-3 SeqFileWriter -