Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # user >> How to check whether chukwa collecting logs


+
Shouvanik Haldar 2013-01-09, 10:54
+
Shouvanik Haldar 2013-01-10, 07:11
+
Eric Yang 2013-01-10, 06:25
Copy link to this message
-
Re: How to check whether chukwa collecting logs
Hi,

In the initial_adaptors, when I mention the below line
add FileTailer /home/ec2-user/test1.log 0

I get a exception like
java.lang.ClassNotFoundException:
org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at
org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor of
class FileTailer

*Now, my question is: *how many arguments are needed? And what is the
purpose of each argument. Having said that,

What does datatype mean in the below example?

add FileTailer FooData /tmp/foo 0

What does "FooData" datatype mean?

I created a sample file named "test1.log". How can I make Chukwa read
from this file?

Please help.

Regards,
Shouvanik
On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:

> Yes, the data is written to HDFS, but you probably need to run demux job
> to organize the data into the form that is suitable for you.
>
> regards,
> Eric
>
>
> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> I am trying to find out a check point , so that I can say that Chukwa is
>> running and collecting logs
>>
>>
>> I configured the /chukwa/etc/chukwa/initial_adaptors
>> with following
>>
>> add FileTailer
>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
>> 0
>>
>> .
>> then I started bin/chukwa agent.
>>
>> Started the hadoop daemons.
>>
>> then, bin/chukwa collector.
>>
>> Next, I check the site
>> "http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true
>> "
>>
>> with following statistics
>>
>> Date:1357728723103
>> Now:1357728765604
>> numberHTTPConnection in time window:17
>> numberchunks in time window:119
>> lifetimechunks:23029
>>
>>
>> and when I check hdfs I see
>>
>> -rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
>> -rw-r--r--   1 ec2-user supergroup     483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
>>
>>
>> -rw-r--r--   1 ec2-user supergroup     483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
>> -rw-r--r--   1 ec2-user supergroup     463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
>>
>>
>> -rw-r--r--   1 ec2-user supergroup     386337 2013-01-08 11:36 /chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done
>> -rw-r--r--   1 ec2-user supergroup     479656 2013-01-08 11:41 /chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done
>>
>>
>> -rw-r--r--   1 ec2-user supergroup     487748 2013-01-08 11:46 /chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done
>> -rw-r--r--   1 ec2-user supergroup     479696 2013-01-08 11:51 /chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done
Thanks,
*Shouvanik*
+
shreyas subramanya 2013-01-10, 07:24
+
Shouvanik Haldar 2013-01-10, 07:34
+
shreyas subramanya 2013-01-10, 07:44
+
Shouvanik Haldar 2013-01-10, 08:06
+
Shouvanik Haldar 2013-01-10, 09:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB