Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa, mail # dev - Need help to configure chukwa to write logs file generated by log4j to hdfs


+
NehaS Singh 2012-12-26, 06:19
+
Eric Yang 2012-12-26, 18:06
+
Neha Singh 2012-12-27, 06:58
Copy link to this message
-
Re: Need help to configure chukwa to write logs file generated by log4j to hdfs
Eric Yang 2012-12-27, 09:06
There doesn't seem to be any errors, nor data received by the collectors.
 Check for errors in agent log file, and make sure there are adaptors
configured to collect data.

regards,
Eric

On Wed, Dec 26, 2012 at 10:58 PM, Neha Singh <[EMAIL PROTECTED]>wrote:

> I have added the HBASE_CONF_DIR as well as HADOOP_CONF_DIR to the classpath
> and my collector.og file looks as below
>
>
> 2012-12-20 16:43:43,919 INFO main ChukwaConfiguration - chukwaConf is
> /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/etc/chukwa
> 2012-12-20 16:43:44,244 INFO main log - Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2012-12-20 16:43:44,288 INFO main log - jetty-6.1.26
> 2012-12-20 16:43:44,331 INFO main root - initing servletCollector
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - rotateInterval is 300000
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - not using fixed time
> interval scheme
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - outputDir is
> /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - fsname is
> hdfs://172.20.104.226:8020
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2012-12-20 16:43:44,816 INFO main log - Started
> SelectChannelConnector@0.0.0.0:9080
> 2012-12-20 16:43:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:43:45,805 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:44:15,805 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:44:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:44:45,805 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:45:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:45:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:45:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:46:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:46:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:46:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:47:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:47:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:47:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:48:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:48:44,805 INFO Timer-2 SeqFileWriter - closing sink
>
> file/home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs/20121220164344752_INFPH01464U_1de46d0$
> 2012-12-20 16:48:44,855 INFO Timer-2 SeqFileWriter - no chunks written to
>
> /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs/20121220164344752_INFPH01464U_1de$
> 2012-12-20 16:48:45,332 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:48:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:49:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:49:45,332 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:49:45,807 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:50:15,806 INFO Timer-3 SeqFileWriter -