Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa, mail # user - How to check whether chukwa collecting logs
Shouvanik Haldar
2013-01-09, 10:54
Eric Yang
2013-01-10, 06:25
Shouvanik Haldar
2013-01-10, 07:11
shreyas subramanya
2013-01-10, 07:24
Shouvanik Haldar
2013-01-10, 07:34
shreyas subramanya
2013-01-10, 07:44
Shouvanik Haldar
2013-01-10, 08:06
Shouvanik Haldar
2013-01-10, 09:42
Copy link to this message
-
Re: How to check whether chukwa collecting logs
Shouvanik Haldar 2013-01-10, 07:01
Hi,

In the initial_adaptors, when I mention the below line
add FileTailer /home/ec2-user/test1.log 0

I get a exception like
java.lang.ClassNotFoundException:
org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at
org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor of
class FileTailer

*Now, my question is: *how many arguments are needed? And what is the
purpose of each argument. Having said that,

What does datatype mean in the below example?

add FileTailer FooData /tmp/foo 0

What does "FooData" datatype mean?

I created a sample file named "test1.log". How can I make Chukwa read
from this file?

Please help.

Regards,
Shouvanik
On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:

> Yes, the data is written to HDFS, but you probably need to run demux job
> to organize the data into the form that is suitable for you.
>
> regards,
> Eric
>
>
> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> I am trying to find out a check point , so that I can say that Chukwa is
>> running and collecting logs
>>
>>
>> I configured the /chukwa/etc/chukwa/initial_adaptors
>> with following
>>
>> add FileTailer
>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
>> 0
>>
>> .
>> then I started bin/chukwa agent.
>>
>> Started the hadoop daemons.
>>
>> then, bin/chukwa collector.
>>
>> Next, I check the site
>> "http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true
>> "
>>
>> with following statistics
>>
>> Date:1357728723103
>> Now:1357728765604
>> numberHTTPConnection in time window:17
>> numberchunks in time window:119
>> lifetimechunks:23029
>>
>>
>> and when I check hdfs I see
>>
>> -rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
>> -rw-r--r--   1 ec2-user supergroup     483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
>>
>>
>> -rw-r--r--   1 ec2-user supergroup     483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
>> -rw-r--r--   1 ec2-user supergroup     463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
>>
>>
>> -rw-r--r--   1 ec2-user supergroup     386337 2013-01-08 11:36 /chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done
>> -rw-r--r--   1 ec2-user supergroup     479656 2013-01-08 11:41 /chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done
>>
>>
>> -rw-r--r--   1 ec2-user supergroup     487748 2013-01-08 11:46 /chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done
>> -rw-r--r--   1 ec2-user supergroup     479696 2013-01-08 11:51 /chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done
Thanks,
*Shouvanik*