Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa, mail # user - How to check whether chukwa collecting logs
Shouvanik Haldar
2013-01-09, 10:54
Shouvanik Haldar
2013-01-10, 07:11
Eric Yang
2013-01-10, 06:25
Shouvanik Haldar
2013-01-10, 07:01
shreyas subramanya
2013-01-10, 07:24
shreyas subramanya
2013-01-10, 07:44
Shouvanik Haldar
2013-01-10, 08:06
Shouvanik Haldar
2013-01-10, 09:42
+
Shouvanik Haldar 2013-01-09, 10:54
+
Shouvanik Haldar 2013-01-10, 07:11
+
Eric Yang 2013-01-10, 06:25
+
Shouvanik Haldar 2013-01-10, 07:01
+
shreyas subramanya 2013-01-10, 07:24
Copy link to this message
-
Re: How to check whether chukwa collecting logs
Shouvanik Haldar 2013-01-10, 07:34
But what does Datatype signify here? Is it programmable.
What does

add FileTailer FooData /tmp/foo 0

mean ? What is "FooData" datatype?

Regards,
Shouvanik
On Thu, Jan 10, 2013 at 12:54 PM, shreyas subramanya
<[EMAIL PROTECTED]>wrote:

> FileTailer is not an adaptor, it is used by the file tailing adapors.
> Use FileTailingAdaptor or one of its subclasses
> - CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.
>
> The general syntax for these adaptors are
> *add FileTailingAdaptor <datatype> [offset-of-first-byte] <path-of-file>
> <offset>*
>
> The adaptor starts tailing from <offset-of-first-byte>
>
> Regards,
> Shreyas
>
>
> On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> In the initial_adaptors, when I mention the below line
>> add FileTailer /home/ec2-user/test1.log 0
>>
>> I get a exception like
>> java.lang.ClassNotFoundException:
>> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>         at java.lang.Class.forName0(Native Method)
>>         at java.lang.Class.forName(Class.java:169)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
>> 2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor of
>> class FileTailer
>>
>> *Now, my question is: *how many arguments are needed? And what is the
>> purpose of each argument. Having said that,
>>
>> What does datatype mean in the below example?
>>
>> add FileTailer FooData /tmp/foo 0
>>
>>
>>
>> What does "FooData" datatype mean?
>>
>> I created a sample file named "test1.log". How can I make Chukwa read from this file?
>>
>> Please help.
>>
>> Regards,
>>
>>
>>
>> Shouvanik
>>
>>
>>
>>
>> On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>
>>> Yes, the data is written to HDFS, but you probably need to run demux job
>>> to organize the data into the form that is suitable for you.
>>>
>>> regards,
>>> Eric
>>>
>>>
>>> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> I am trying to find out a check point , so that I can say that Chukwa
>>>> is running and collecting logs
>>>>
>>>>
>>>> I configured the /chukwa/etc/chukwa/initial_adaptors
>>>> with following
>>>>
>>>> add FileTailer
>>>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
>>>> 0
>>>>
>>>> .
>>>> then I started bin/chukwa agent.
>>>>
>>>> Started the hadoop daemons.
>>>>
>>>> then, bin/chukwa collector.
>>>>
>>>> Next, I check the site
>>>> "
>>>> http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true
>>>> "
>>>>
>>>> with following statistics
>>>>
>>>> Date:1357728723103
>>>> Now:1357728765604
>>>> numberHTTPConnection in time window:17
>>>> numberchunks in time window:119
>>>> lifetimechunks:23029
>>>>
>>>>
>>>> and when I check hdfs I see
>>>>
>>>> -rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
Thanks,
*Shouvanik*
+
shreyas subramanya 2013-01-10, 07:44
+
Shouvanik Haldar 2013-01-10, 08:06
+
Shouvanik Haldar 2013-01-10, 09:42