Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa, mail # user - How to check whether chukwa collecting logs
Shouvanik Haldar
2013-01-09, 10:54
Shouvanik Haldar
2013-01-10, 07:11
Eric Yang
2013-01-10, 06:25
Shouvanik Haldar
2013-01-10, 07:01
Shouvanik Haldar
2013-01-10, 07:34
shreyas subramanya
2013-01-10, 07:44
Shouvanik Haldar
2013-01-10, 08:06
Shouvanik Haldar
2013-01-10, 09:42
+
Shouvanik Haldar 2013-01-09, 10:54
+
Shouvanik Haldar 2013-01-10, 07:11
+
Eric Yang 2013-01-10, 06:25
+
Shouvanik Haldar 2013-01-10, 07:01
Copy link to this message
-
Re: How to check whether chukwa collecting logs
shreyas subramanya 2013-01-10, 07:24
FileTailer is not an adaptor, it is used by the file tailing adapors.
Use FileTailingAdaptor or one of its subclasses
- CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.

The general syntax for these adaptors are
*add FileTailingAdaptor <datatype> [offset-of-first-byte] <path-of-file>
<offset>*

The adaptor starts tailing from <offset-of-first-byte>

Regards,
Shreyas
On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> In the initial_adaptors, when I mention the below line
> add FileTailer /home/ec2-user/test1.log 0
>
> I get a exception like
> java.lang.ClassNotFoundException:
> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:169)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
> 2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor of
> class FileTailer
>
> *Now, my question is: *how many arguments are needed? And what is the
> purpose of each argument. Having said that,
>
> What does datatype mean in the below example?
>
> add FileTailer FooData /tmp/foo 0
>
>
> What does "FooData" datatype mean?
>
> I created a sample file named "test1.log". How can I make Chukwa read from this file?
>
> Please help.
>
> Regards,
>
>
> Shouvanik
>
>
>
>
> On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> Yes, the data is written to HDFS, but you probably need to run demux job
>> to organize the data into the form that is suitable for you.
>>
>> regards,
>> Eric
>>
>>
>> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
>> [EMAIL PROTECTED]> wrote:
>>
>>> I am trying to find out a check point , so that I can say that Chukwa is
>>> running and collecting logs
>>>
>>>
>>> I configured the /chukwa/etc/chukwa/initial_adaptors
>>> with following
>>>
>>> add FileTailer
>>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
>>> 0
>>>
>>> .
>>> then I started bin/chukwa agent.
>>>
>>> Started the hadoop daemons.
>>>
>>> then, bin/chukwa collector.
>>>
>>> Next, I check the site
>>> "
>>> http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true
>>> "
>>>
>>> with following statistics
>>>
>>> Date:1357728723103
>>> Now:1357728765604
>>> numberHTTPConnection in time window:17
>>> numberchunks in time window:119
>>> lifetimechunks:23029
>>>
>>>
>>> and when I check hdfs I see
>>>
>>> -rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
>>> -rw-r--r--   1 ec2-user supergroup     483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
>>>
>>>
>>>
>>>
>>> -rw-r--r--   1 ec2-user supergroup     483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
>>> -rw-r--r--   1 ec2-user supergroup     463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
+
Shouvanik Haldar 2013-01-10, 07:34
+
shreyas subramanya 2013-01-10, 07:44
+
Shouvanik Haldar 2013-01-10, 08:06
+
Shouvanik Haldar 2013-01-10, 09:42