Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # user >> How to check whether chukwa collecting logs


+
Shouvanik Haldar 2013-01-09, 10:54
+
Shouvanik Haldar 2013-01-10, 07:11
+
Eric Yang 2013-01-10, 06:25
+
Shouvanik Haldar 2013-01-10, 07:01
Copy link to this message
-
Re: How to check whether chukwa collecting logs
FileTailer is not an adaptor, it is used by the file tailing adapors.
Use FileTailingAdaptor or one of its subclasses
- CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.

The general syntax for these adaptors are
*add FileTailingAdaptor <datatype> [offset-of-first-byte] <path-of-file>
<offset>*

The adaptor starts tailing from <offset-of-first-byte>

Regards,
Shreyas
On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> In the initial_adaptors, when I mention the below line
> add FileTailer /home/ec2-user/test1.log 0
>
> I get a exception like
> java.lang.ClassNotFoundException:
> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:169)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
> 2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor of
> class FileTailer
>
> *Now, my question is: *how many arguments are needed? And what is the
> purpose of each argument. Having said that,
>
> What does datatype mean in the below example?
>
> add FileTailer FooData /tmp/foo 0
>
>
> What does "FooData" datatype mean?
>
> I created a sample file named "test1.log". How can I make Chukwa read from this file?
>
> Please help.
>
> Regards,
>
>
> Shouvanik
>
>
>
>
> On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> Yes, the data is written to HDFS, but you probably need to run demux job
>> to organize the data into the form that is suitable for you.
>>
>> regards,
>> Eric
>>
>>
>> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
>> [EMAIL PROTECTED]> wrote:
>>
>>> I am trying to find out a check point , so that I can say that Chukwa is
>>> running and collecting logs
>>>
>>>
>>> I configured the /chukwa/etc/chukwa/initial_adaptors
>>> with following
>>>
>>> add FileTailer
>>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
>>> 0
>>>
>>> .
>>> then I started bin/chukwa agent.
>>>
>>> Started the hadoop daemons.
>>>
>>> then, bin/chukwa collector.
>>>
>>> Next, I check the site
>>> "
>>> http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true
>>> "
>>>
>>> with following statistics
>>>
>>> Date:1357728723103
>>> Now:1357728765604
>>> numberHTTPConnection in time window:17
>>> numberchunks in time window:119
>>> lifetimechunks:23029
>>>
>>>
>>> and when I check hdfs I see
>>>
>>> -rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
>>> -rw-r--r--   1 ec2-user supergroup     483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
>>>
>>>
>>>
>>>
>>> -rw-r--r--   1 ec2-user supergroup     483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
>>> -rw-r--r--   1 ec2-user supergroup     463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
+
Shouvanik Haldar 2013-01-10, 07:34
+
shreyas subramanya 2013-01-10, 07:44
+
Shouvanik Haldar 2013-01-10, 08:06
+
Shouvanik Haldar 2013-01-10, 09:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB