Chukwa, mail # user - Re: How to check whether chukwa collecting logs

How to check whether chukwa collecting logs

Shouvanik Haldar 2013-01-09, 10:54

I am trying to find out a check point , so that I can say that Chukwa is
running and collecting logs
I configured the /chukwa/etc/chukwa/initial_adaptors
with following

add FileTailer
/home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
0

.
then I started bin/chukwa agent.

Started the hadoop daemons.

then, bin/chukwa collector.

Next, I check the site
"http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true"

with following statistics

Date:1357728723103
Now:1357728765604
numberHTTPConnection in time window:17
numberchunks in time window:119
lifetimechunks:23029
and when I check hdfs I see

-rw-r--r-- 1 ec2-user supergroup 657818 2013-01-08 11:09
/chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
-rw-r--r-- 1 ec2-user supergroup 483520 2013-01-08 11:14
/chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
-rw-r--r-- 1 ec2-user supergroup 483155 2013-01-08 11:19
/chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
-rw-r--r-- 1 ec2-user supergroup 463708 2013-01-08 11:24
/chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
-rw-r--r-- 1 ec2-user supergroup 386337 2013-01-08 11:36
/chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done
-rw-r--r-- 1 ec2-user supergroup 479656 2013-01-08 11:41
/chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done
-rw-r--r-- 1 ec2-user supergroup 487748 2013-01-08 11:46
/chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done
-rw-r--r-- 1 ec2-user supergroup 479696 2013-01-08 11:51
/chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done
-rw-r--r-- 1 ec2-user supergroup 488030 2013-01-08 11:56
/chukwa/logs/201308115135034_domU12313907193B_488afeb413c19ee05bf7ff5.done
-rw-r--r-- 1 ec2-user supergroup 479754 2013-01-08 12:01
/chukwa/logs/201308115635034_domU12313907193B_488afeb413c19ee05bf7ff2.done
-rw-r--r-- 1 ec2-user supergroup 488067 2013-01-08 12:06
/chukwa/logs/201308120135034_domU12313907193B_488afeb413c19ee05bf7fef.done
-rw-r--r-- 1 ec2-user supergroup 479961 2013-01-08 12:11
/chukwa/logs/201308120635034_domU12313907193B_488afeb413c19ee05bf7fec.done
-rw-r--r-- 1 ec2-user supergroup 487976 2013-01-08 12:16
/chukwa/logs/201308121135034_domU12313907193B_488afeb413c19ee05bf7fe9.done
-rw-r--r-- 1 ec2-user supergroup 479863 2013-01-08 12:21
/chukwa/logs/201308121635034_domU12313907193B_488afeb413c19ee05bf7fe6.done
-rw-r--r-- 1 ec2-user supergroup 649168 2013-01-08 12:42
/chukwa/logs/201308123755390_domU12313907193B_5ab1511313c1a2ac24f8000.done
-rw-r--r-- 1 ec2-user supergroup 483972 2013-01-08 12:47
/chukwa/logs/201308124255466_domU12313907193B_5ab1511313c1a2ac24f7ffc.done
-rw-r--r-- 1 ec2-user supergroup 483417 2013-01-08 12:52
/chukwa/logs/201308124755466_domU12313907193B_5ab1511313c1a2ac24f7ff8.done
-rw-r--r-- 1 ec2-user supergroup 576711 2013-01-08 13:06
/chukwa/logs/201308130134839_domU12313907193B_2abeddbf13c1a406b0b8000.done
-rw-r--r-- 1 ec2-user supergroup 483877 2013-01-08 13:11
/chukwa/logs/201308130634912_domU12313907193B_2abeddbf13c1a406b0b7ffc.done
-rw-r--r-- 1 ec2-user supergroup 484352 2013-01-08 13:16
/chukwa/logs/201308131134912_domU12313907193B_2abeddbf13c1a406b0b7ff7.done
-rw-r--r-- 1 ec2-user supergroup 486158 2013-01-08 13:21
/chukwa/logs/201308131634912_domU12313907193B_2abeddbf13c1a406b0b7ff2.done
-rw-r--r-- 1 ec2-user supergroup 483925 2013-01-08 13:26
/chukwa/logs/201308132134913_domU12313907193B_2abeddbf13c1a406b0b7fed.done
-rw-r--r-- 1 ec2-user supergroup 629788 2013-01-08 13:37
/chukwa/logs/201308133253308_domU12313907193B_7382697b13c1a5d14db8000.done
-rw-r--r-- 1 ec2-user supergroup 485716 2013-01-08 13:42
/chukwa/logs/201308133753394_domU12313907193B_7382697b13c1a5d14db7ffb.done
-rw-r--r-- 1 ec2-user supergroup 486475 2013-01-08 13:47
/chukwa/logs/201308134253394_domU12313907193B_7382697b13c1a5d14db7ff5.done
-rw-r--r-- 1 ec2-user supergroup 481330 2013-01-08 13:52
/chukwa/logs/201308134753394_domU12313907193B_7382697b13c1a5d14db7fef.done
-rw-r--r-- 1 ec2-user supergroup 488138 2013-01-08 13:57
/chukwa/logs/201308135253394_domU12313907193B_7382697b13c1a5d14db7fe9.done
-rw-r--r-- 1 ec2-user supergroup 484294 2013-01-08 14:02
/chukwa/logs/201308135753394_domU12313907193B_7382697b13c1a5d14db7fe3.done
-rw-r--r-- 1 ec2-user supergroup 480689 2013-01-08 14:07
/chukwa/logs/201308140253394_domU12313907193B_7382697b13c1a5d14db7fdd.done
-rw-r--r-- 1 ec2-user supergroup 488018 2013-01-08 14:12
/chukwa/logs/201308140753395_domU12313907193B_7382697b13c1a5d14db7fd7.done
-rw-r--r-- 1 ec2-user supergroup 287583 2013-01-08 14:23
/chukwa/logs/201308141253395_domU12313907193B_7382697b13c1a5d14db7fd1.done
-rw-r--r-- 1 ec2-user supergroup 527781 2013-01-08 14:27
/chukwa/logs/201308142254785_domU12313907193B_3065fa8513c1a8ae1598000.done
-rw-r--r-- 1 ec2-user supergroup 483844 2013-01-08 14:32
/chukwa/logs/201308142754897_domU12313907193B_3065fa8513c1a8ae1597ff9.done
-rw-r--r-- 1 ec2-user supergroup 483447 2013-01-08 14:37
/chukwa/logs/201308143254897_domU12313907193B_3065fa8513c1a8ae1597ff3.done
-rw-r--r-- 1 ec2-user supergroup 483915 2013-01-08 14:42
/chukwa/logs/201308143754897_domU12313907193B_3065fa8513c1a8ae1597fed.done
-rw-r--r-- 1 ec2-user supergroup 483902 2013-01-08 14:47
/chukwa/logs/201308144254897_domU12313907193B_3065fa8513c1a8ae1597fe7.done
-rw-r--r-- 1 ec2-user supergroup 484187 2013-01-08 14:52
/chukwa/logs/201308144754897_domU12313907193B_3065fa8513c1a8ae1597fe1.done
-rw-r--r-- 1 ec2-user supergroup 295291 2013-01-08 17:04
/chukwa/logs/201308145254897_domU12313907193B_3065fa8513c1a8ae1597fdb.done
-rw-r--r-- 1 ec2-user supergroup 524107 2013-01-08 17:08
/chukwa/logs/201308170310783_domU12313

Re: How to check whether chukwa collecting logs

Shouvanik Haldar 2013-01-10, 07:11

Can you please help me find out why the following log is generated by the
collector

2013-01-10 07:09:23,864 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:18,numberchunks:166
2013-01-10 07:09:23,884 WARN LocalToRemoteHdfsMover CopySequenceFile -
Error during .chukwa file recovery
java.io.EOFException
   at java.io.DataInputStream.readFully(DataInputStream.java:180)
   at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
   at
org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
   at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1988)
   at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1888)
   at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1934)
   at
org.apache.hadoop.chukwa.util.CopySequenceFile.createValidSequenceFile(CopySequenceFile.java:78)
   at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.cleanup(LocalToRemoteHdfsMover.java:186)
   at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.run(LocalToRemoteHdfsMover.java:216)
2013-01-10 07:09:23,891 INFO LocalToRemoteHdfsMover LocalToRemoteHdfsMover
- Deleted .recover file,
/tmp/chukwa/dataSink/201308112441296_domU12313907193B_216836a313c19d565ee7ff8.recover
2013-01-10 07:09:23,891 INFO LocalToRemoteHdfsMover LocalToRemoteHdfsMover
- Copying .chukwa file to valid sink file before moving,
/tmp/chukwa/dataSink/201310063103378_domU1231390B80BA_1082291f13c22c778057f42.chukwa
2013-01-10 07:09:23,899 WARN LocalToRemoteHdfsMover FSInputChecker -
Problem opening checksum file:
/tmp/chukwa/dataSink/201310063103378_domU1231390B80BA_1082291f13c22c778057f42.chukwa.
Ignoring exception: java.io.EOFException
   at java.io.DataInputStream.readFully(DataInputStream.java:180)
   at java.io.DataInputStream.readFully(DataInputStream.java:152)
   at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:134)
   at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
   at
org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1495)
   at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1482)
   at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
   at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
   at
org.apache.hadoop.chukwa.util.CopySequenceFile.createValidSequenceFile(CopySequenceFile.java:73)
   at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.cleanup(LocalToRemoteHdfsMover.java:186)
   at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.run(LocalToRemoteHdfsMover.java:216)

2013-01-10 07:09:23,931 WARN LocalToRemoteHdfsMover CopySequenceFile -
Error during .chukwa file recovery
java.io.EOFException
   at java.io.DataInputStream.readFully(DataInputStream.java:180)
   at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
   at
org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
   at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1988)
   at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1888)
   at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1934)
   at
org.apache.hadoop.chukwa.util.CopySequenceFile.createValidSequenceFile(CopySequenceFile.java:78)
   at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.cleanup(LocalToRemoteHdfsMover.java:186)
   at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.run(LocalToRemoteHdfsMover.java:216)
2013-01-10 07:09:24,348 INFO Timer-3 LocalWriter -
stat:datacollection.writer.local.LocalWriter dataSize=29534 dataRate=984
2013-01-10 07:09:54,349 INFO Timer-3 LocalWriter -
stat:datacollection.writer.local.LocalWriter dataSize=34440 dataRate=1147
Regards,
Shouvanik
On Wed, Jan 9, 2013 at 4:24 PM, Shouvanik Haldar <[EMAIL PROTECTED]
Thanks,
*Shouvanik*

Re: How to check whether chukwa collecting logs

Eric Yang 2013-01-10, 06:25

Yes, the data is written to HDFS, but you probably need to run demux job to
organize the data into the form that is suitable for you.

regards,
Eric

On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <[EMAIL PROTECTED]
> wrote:

> I am trying to find out a check point , so that I can say that Chukwa is
> running and collecting logs
>
>
> I configured the /chukwa/etc/chukwa/initial_adaptors
> with following
>
> add FileTailer
> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
> 0
>
> .
> then I started bin/chukwa agent.
>
> Started the hadoop daemons.
>
> then, bin/chukwa collector.
>
> Next, I check the site
> "http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true"
>
> with following statistics
>
> Date:1357728723103
> Now:1357728765604
> numberHTTPConnection in time window:17
> numberchunks in time window:119
> lifetimechunks:23029
>
>
> and when I check hdfs I see
>
> -rw-r--r-- 1 ec2-user supergroup 657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
> -rw-r--r-- 1 ec2-user supergroup 483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
>
> -rw-r--r-- 1 ec2-user supergroup 483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
> -rw-r--r-- 1 ec2-user supergroup 463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
>
> -rw-r--r-- 1 ec2-user supergroup 386337 2013-01-08 11:36 /chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done
> -rw-r--r-- 1 ec2-user supergroup 479656 2013-01-08 11:41 /chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done
>
> -rw-r--r-- 1 ec2-user supergroup 487748 2013-01-08 11:46 /chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done
> -rw-r--r-- 1 ec2-user supergroup 479696 2013-01-08 11:51 /chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done
>
> -rw-r--r-- 1 ec2-user supergroup 488030 2013-01-08 11:56 /chukwa/logs/201308115135034_domU12313907193B_488afeb413c19ee05bf7ff5.done
> -rw-r--r-- 1 ec2-user supergroup 479754 2013-01-08 12:01 /chukwa/logs/201308115635034_domU12313907193B_488afeb413c19ee05bf7ff2.done
>
> -rw-r--r-- 1 ec2-user supergroup 488067 2013-01-08 12:06 /chukwa/logs/201308120135034_domU12313907193B_488afeb413c19ee05bf7fef.done
> -rw-r--r-- 1 ec2-user supergroup 479961 2013-01-08 12:11 /chukwa/logs/201308120635034_domU12313907193B_488afeb413c19ee05bf7fec.done
>
> -rw-r--r-- 1 ec2-user supergroup 487976 2013-01-08 12:16 /chukwa/logs/201308121135034_domU12313907193B_488afeb413c19ee05bf7fe9.done
> -rw-r--r-- 1 ec2-user supergroup 479863 2013-01-08 12:21 /chukwa/logs/201308121635034_domU12313907193B_488afeb413c19ee05bf7fe6.done
>
> -rw-r--r-- 1 ec2-user supergroup 649168 2013-01-08 12:42 /chukwa/logs/201308123755390_domU12313907193B_5ab1511313c1a2ac24f8000.done
> -rw-r--r-- 1 ec2-user supergroup 483972 2013-01-08 12:47 /chukwa/logs/201308124255466_domU12313907193B_5ab1511313c1a2ac24f7ffc.done
>
> -rw-r--r-- 1 ec2-user supergroup 483417 2013-01-08 12:52 /chukwa/logs/201308124755466_domU12313907193B_5ab1511313c1a2ac24f7ff8.done
> -rw-r--r-- 1 ec2-user supergroup 576711 2013-01-08 13:06 /chukwa/logs/201308130134839_domU12313907193B_2abeddbf13c1a406b0b8000.done
>
> -rw-r--r-- 1 ec2-user supergroup 483877 2013-01-08 13:11 /chukwa/logs/201308130634912_domU12313907193B_2abeddbf13c1a406b0b7ffc.done
> -rw-r--r-- 1 ec2-user supergroup 484352 2013-01-08 13:16 /chukwa/logs/201308131134912_domU12313907193B_2abeddbf13c1a406b0b7ff7.done
>
> -rw-r--r-- 1 ec2-user supergroup 486158 2013-01-08 13:21 /chukwa/logs/201308131634912_domU12313907193B_2abeddbf13c1a406b0b7ff2.done
> -rw-r--r-- 1 ec2-user supergroup 483925 2013-01-08 13:26 /chukwa/logs/201308132134913_domU12313907193B_2abeddbf13c1a406b0b7fed.done

Re: How to check whether chukwa collecting logs

Shouvanik Haldar 2013-01-10, 07:01

Hi,

In the initial_adaptors, when I mention the below line
add FileTailer /home/ec2-user/test1.log 0

I get a exception like
java.lang.ClassNotFoundException:
org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Class.java:169)
   at
org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
   at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
   at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
   at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
   at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
   at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor of
class FileTailer

*Now, my question is: *how many arguments are needed? And what is the
purpose of each argument. Having said that,

What does datatype mean in the below example?

add FileTailer FooData /tmp/foo 0

What does "FooData" datatype mean?

I created a sample file named "test1.log". How can I make Chukwa read
from this file?

Please help.

Regards,
Shouvanik
On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:

> Yes, the data is written to HDFS, but you probably need to run demux job
> to organize the data into the form that is suitable for you.
>
> regards,
> Eric
>
>
> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> I am trying to find out a check point , so that I can say that Chukwa is
>> running and collecting logs
>>
>>
>> I configured the /chukwa/etc/chukwa/initial_adaptors
>> with following
>>
>> add FileTailer
>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
>> 0
>>
>> .
>> then I started bin/chukwa agent.
>>
>> Started the hadoop daemons.
>>
>> then, bin/chukwa collector.
>>
>> Next, I check the site
>> "http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true
>> "
>>
>> with following statistics
>>
>> Date:1357728723103
>> Now:1357728765604
>> numberHTTPConnection in time window:17
>> numberchunks in time window:119
>> lifetimechunks:23029
>>
>>
>> and when I check hdfs I see
>>
>> -rw-r--r-- 1 ec2-user supergroup 657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
>> -rw-r--r-- 1 ec2-user supergroup 483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
>>
>>
>> -rw-r--r-- 1 ec2-user supergroup 483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
>> -rw-r--r-- 1 ec2-user supergroup 463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
>>
>>
>> -rw-r--r-- 1 ec2-user supergroup 386337 2013-01-08 11:36 /chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done
>> -rw-r--r-- 1 ec2-user supergroup 479656 2013-01-08 11:41 /chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done
>>
>>
>> -rw-r--r-- 1 ec2-user supergroup 487748 2013-01-08 11:46 /chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done
>> -rw-r--r-- 1 ec2-user supergroup 479696 2013-01-08 11:51 /chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done
Thanks,
*Shouvanik*

Re: How to check whether chukwa collecting logs

shreyas subramanya 2013-01-10, 07:24

FileTailer is not an adaptor, it is used by the file tailing adapors.
Use FileTailingAdaptor or one of its subclasses
- CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.

The general syntax for these adaptors are
*add FileTailingAdaptor <datatype> [offset-of-first-byte] <path-of-file>
<offset>*

The adaptor starts tailing from <offset-of-first-byte>

Regards,
Shreyas
On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> In the initial_adaptors, when I mention the below line
> add FileTailer /home/ec2-user/test1.log 0
>
> I get a exception like
> java.lang.ClassNotFoundException:
> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:169)
> at
> org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
> at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
> at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
> at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
> at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
> at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
> 2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor of
> class FileTailer
>
> *Now, my question is: *how many arguments are needed? And what is the
> purpose of each argument. Having said that,
>
> What does datatype mean in the below example?
>
> add FileTailer FooData /tmp/foo 0
>
>
> What does "FooData" datatype mean?
>
> I created a sample file named "test1.log". How can I make Chukwa read from this file?
>
> Please help.
>
> Regards,
>
>
> Shouvanik
>
>
>
>
> On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> Yes, the data is written to HDFS, but you probably need to run demux job
>> to organize the data into the form that is suitable for you.
>>
>> regards,
>> Eric
>>
>>
>> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
>> [EMAIL PROTECTED]> wrote:
>>
>>> I am trying to find out a check point , so that I can say that Chukwa is
>>> running and collecting logs
>>>
>>>
>>> I configured the /chukwa/etc/chukwa/initial_adaptors
>>> with following
>>>
>>> add FileTailer
>>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
>>> 0
>>>
>>> .
>>> then I started bin/chukwa agent.
>>>
>>> Started the hadoop daemons.
>>>
>>> then, bin/chukwa collector.
>>>
>>> Next, I check the site
>>> "
>>> http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true
>>> "
>>>
>>> with following statistics
>>>
>>> Date:1357728723103
>>> Now:1357728765604
>>> numberHTTPConnection in time window:17
>>> numberchunks in time window:119
>>> lifetimechunks:23029
>>>
>>>
>>> and when I check hdfs I see
>>>
>>> -rw-r--r-- 1 ec2-user supergroup 657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
>>> -rw-r--r-- 1 ec2-user supergroup 483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
>>>
>>>
>>>
>>>
>>> -rw-r--r-- 1 ec2-user supergroup 483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
>>> -rw-r--r-- 1 ec2-user supergroup 463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done

Re: How to check whether chukwa collecting logs

Shouvanik Haldar 2013-01-10, 07:34

But what does Datatype signify here? Is it programmable.
What does

add FileTailer FooData /tmp/foo 0

mean ? What is "FooData" datatype?

Regards,
Shouvanik
On Thu, Jan 10, 2013 at 12:54 PM, shreyas subramanya
<[EMAIL PROTECTED]>wrote:

> FileTailer is not an adaptor, it is used by the file tailing adapors.
> Use FileTailingAdaptor or one of its subclasses
> - CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.
>
> The general syntax for these adaptors are
> *add FileTailingAdaptor <datatype> [offset-of-first-byte] <path-of-file>
> <offset>*
>
> The adaptor starts tailing from <offset-of-first-byte>
>
> Regards,
> Shreyas
>
>
> On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> In the initial_adaptors, when I mention the below line
>> add FileTailer /home/ec2-user/test1.log 0
>>
>> I get a exception like
>> java.lang.ClassNotFoundException:
>> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:169)
>> at
>> org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
>> at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
>> at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
>> at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
>> at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
>> at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
>> 2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor of
>> class FileTailer
>>
>> *Now, my question is: *how many arguments are needed? And what is the
>> purpose of each argument. Having said that,
>>
>> What does datatype mean in the below example?
>>
>> add FileTailer FooData /tmp/foo 0
>>
>>
>>
>> What does "FooData" datatype mean?
>>
>> I created a sample file named "test1.log". How can I make Chukwa read from this file?
>>
>> Please help.
>>
>> Regards,
>>
>>
>>
>> Shouvanik
>>
>>
>>
>>
>> On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>
>>> Yes, the data is written to HDFS, but you probably need to run demux job
>>> to organize the data into the form that is suitable for you.
>>>
>>> regards,
>>> Eric
>>>
>>>
>>> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> I am trying to find out a check point , so that I can say that Chukwa
>>>> is running and collecting logs
>>>>
>>>>
>>>> I configured the /chukwa/etc/chukwa/initial_adaptors
>>>> with following
>>>>
>>>> add FileTailer
>>>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
>>>> 0
>>>>
>>>> .
>>>> then I started bin/chukwa agent.
>>>>
>>>> Started the hadoop daemons.
>>>>
>>>> then, bin/chukwa collector.
>>>>
>>>> Next, I check the site
>>>> "
>>>> http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true
>>>> "
>>>>
>>>> with following statistics
>>>>
>>>> Date:1357728723103
>>>> Now:1357728765604
>>>> numberHTTPConnection in time window:17
>>>> numberchunks in time window:119
>>>> lifetimechunks:23029
>>>>
>>>>
>>>> and when I check hdfs I see
>>>>
>>>> -rw-r--r-- 1 ec2-user supergroup 657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
Thanks,
*Shouvanik*

Re: How to check whether chukwa collecting logs

shreyas subramanya 2013-01-10, 07:44

Datatype is part of the metadata for a chunk. How or whether it is used
depends on the ChukwaWriter that is configured in the collecotr. For
example, in HBaseWriter, the datatype is used to figure out which processor
class to invoke, which in-turn decides which hbase table to write the
particular chunk data to. For a SeqFileWriter, it is just serialized in the
record key. HTH.

Regards,
Shreyas

On Wed, Jan 9, 2013 at 11:34 PM, Shouvanik Haldar <
[EMAIL PROTECTED]> wrote:

> But what does Datatype signify here? Is it programmable.
> What does
>
>
> add FileTailer FooData /tmp/foo 0
>
> mean ? What is "FooData" datatype?
>
> Regards,
>
> Shouvanik
>
>
>
>
> On Thu, Jan 10, 2013 at 12:54 PM, shreyas subramanya <
> [EMAIL PROTECTED]> wrote:
>
>> FileTailer is not an adaptor, it is used by the file tailing adapors.
>> Use FileTailingAdaptor or one of its subclasses
>> - CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.
>>
>> The general syntax for these adaptors are
>> *add FileTailingAdaptor <datatype> [offset-of-first-byte] <path-of-file>
>> <offset>*
>>
>> The adaptor starts tailing from <offset-of-first-byte>
>>
>> Regards,
>> Shreyas
>>
>>
>> On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> In the initial_adaptors, when I mention the below line
>>> add FileTailer /home/ec2-user/test1.log 0
>>>
>>> I get a exception like
>>> java.lang.ClassNotFoundException:
>>> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:169)
>>> at
>>> org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
>>> at
>>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
>>> at
>>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
>>> at
>>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
>>> at
>>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
>>> at
>>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
>>> 2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor
>>> of class FileTailer
>>>
>>> *Now, my question is: *how many arguments are needed? And what is the
>>> purpose of each argument. Having said that,
>>>
>>> What does datatype mean in the below example?
>>>
>>> add FileTailer FooData /tmp/foo 0
>>>
>>>
>>>
>>>
>>>
>>> What does "FooData" datatype mean?
>>>
>>> I created a sample file named "test1.log". How can I make Chukwa read from this file?
>>>
>>> Please help.
>>>
>>> Regards,
>>>
>>>
>>>
>>>
>>>
>>> Shouvanik
>>>
>>>
>>>
>>>
>>> On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>>
>>>> Yes, the data is written to HDFS, but you probably need to run demux
>>>> job to organize the data into the form that is suitable for you.
>>>>
>>>> regards,
>>>> Eric
>>>>
>>>>
>>>> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> I am trying to find out a check point , so that I can say that Chukwa
>>>>> is running and collecting logs
>>>>>
>>>>>
>>>>> I configured the /chukwa/etc/chukwa/initial_adaptors
>>>>> with following
>>>>>
>>>>> add FileTailer
>>>>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log

Re: How to check whether chukwa collecting logs

Shouvanik Haldar 2013-01-10, 08:06

Thanks.

I want to run Map Reduce Job on the .done files in hdfs. I have used the
following command

bin/chukwa dumpArchive
hdfs://localhost:9000/chukwa/logs/201310074423346_domU1231390B80BA_675a448513c23461dfb7fa8.done

The output shows a file which contains data in a structured way like

==============Cluster : cluster="chukwa"
DataType : HadoopMetrics
Source : domU-12-31-39-0B-80-BA
Application : domU-12-31-39-0B-80-BA
SeqID : 488
Data : 2013-01-10 07:44:33,021 INFO chukwa.metrics.HadoopMetrics:
{"ScheduledReplicationBlocks":0,"BlocksTotal":131,"recordName":"FSNamesystem","CapacityTotalGB":8,"CapacityUsedGB":0,"ExcessBlocks":0,"MissingBlocks":0,"PendingReplicationBlocks":0,"timestamp":1357803873021,"FilesTotal":143,"CapacityRemainingGB":6,"UnderReplicatedBlocks":0,"context":"dfs","hostName":"domU-12-31-39-0B-80-BA","TotalLoad":1,"PendingDeletionBlocks":0,"contextName":"dfs","CorruptBlocks":0,"BlockCapacity":2097152}
TimePartition: 1357801200000
DataType: HadoopMetrics
StreamName: cluster="chukwa"/domU-12-31-39-0B-80-BA/domU-12-31-39-0B-80-BA
SeqId: 553
==============Cluster : cluster="chukwa"
DataType : HadoopMetrics
Source : domU-12-31-39-0B-80-BA
Application : domU-12-31-39-0B-80-BA
SeqID : 553
Data : 2013-01-10 07:44:33,022 INFO chukwa.metrics.HadoopMetrics:
{"port":"9000","getFileInfo_avg_time":0.0,"addBlock_num_ops":9,"rename_avg_time":3.0,"addBlock_avg_time":0.0,"recordName":"rpcdetailed","sendHeartbeat_num_ops":825,"rename_num_ops":8,"blockReceived_avg_time":0.0,"create_avg_time":3.0,"getFileInfo_num_ops":19,"timestamp":1357803873021,"sendHeartbeat_avg_time":0.0,"complete_avg_time":12.0,"complete_num_ops":9,"create_num_ops":9,"blockReceived_num_ops

Does it mean, that Map Reduce job has run?

Please help

Regards,
Shouvanik
On Thu, Jan 10, 2013 at 1:14 PM, shreyas subramanya
<[EMAIL PROTECTED]>wrote:

> Datatype is part of the metadata for a chunk. How or whether it is used
> depends on the ChukwaWriter that is configured in the collecotr. For
> example, in HBaseWriter, the datatype is used to figure out which processor
> class to invoke, which in-turn decides which hbase table to write the
> particular chunk data to. For a SeqFileWriter, it is just serialized in the
> record key. HTH.
>
> Regards,
> Shreyas
>
>
> On Wed, Jan 9, 2013 at 11:34 PM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> But what does Datatype signify here? Is it programmable.
>> What does
>>
>>
>> add FileTailer FooData /tmp/foo 0
>>
>> mean ? What is "FooData" datatype?
>>
>> Regards,
>>
>> Shouvanik
>>
>>
>>
>>
>> On Thu, Jan 10, 2013 at 12:54 PM, shreyas subramanya <
>> [EMAIL PROTECTED]> wrote:
>>
>>> FileTailer is not an adaptor, it is used by the file tailing adapors.
>>> Use FileTailingAdaptor or one of its subclasses
>>> - CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.
>>>
>>> The general syntax for these adaptors are
>>> *add FileTailingAdaptor <datatype> [offset-of-first-byte]
>>> <path-of-file> <offset>*
>>>
>>> The adaptor starts tailing from <offset-of-first-byte>
>>>
>>> Regards,
>>> Shreyas
>>>
>>>
>>> On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>
>>>> In the initial_adaptors, when I mention the below line
>>>> add FileTailer /home/ec2-user/test1.log 0
>>>>
>>>> I get a exception like
>>>> java.lang.ClassNotFoundException:
>>>> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>> at java.lang.Class.forName0(Native Method)
>>>> at java.lang.Class.forName(Class.java:169)
Thanks,
*Shouvanik*

Re: How to check whether chukwa collecting logs

Shouvanik Haldar 2013-01-10, 09:42

* I am not able to start HICC. After logging in with "admin"/"admin",

*
*getting the exception

2013-01-10 09:31:21,030 ERROR 193738361@qtp-1330491619-7 log - /hicc/
java.lang.NoSuchFieldError: STANDARD_IMPORTS
   at org.apache.jasper.compiler.PageInfo.<init>(PageInfo.java:101)
   at
org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:108)
   at org.apache.jasper.compiler.Compiler.compile(Compiler.java:296)
   at org.apache.jasper.compiler.Compiler.compile(Compiler.java:277)
   at org.apache.jasper.compiler.Compiler.compile(Compiler.java:265)
   at
org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:564)
   at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:299)
   at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:315)
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
   at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
   at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
   at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
   at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
   at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
   at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
   at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
   at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
   at
org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
   at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
   at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
   at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
   at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
   at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
   at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
   at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
   at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
   at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
   at org.mortbay.jetty.Server.handle(Server.java:326)
   at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
   at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
   at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
   at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
   at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
   at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
   at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

*
*Please help.
*
On Thu, Jan 10, 2013 at 1:36 PM, Shouvanik Haldar <
[EMAIL PROTECTED]> wrote:

> Thanks.
>
> I want to run Map Reduce Job on the .done files in hdfs. I have used the
> following command
>
> bin/chukwa dumpArchive
> hdfs://localhost:9000/chukwa/logs/201310074423346_domU1231390B80BA_675a448513c23461dfb7fa8.done
>
> The output shows a file which contains data in a structured way like
>
> ==============> Cluster : cluster="chukwa"
> DataType : HadoopMetrics
> Source : domU-12-31-39-0B-80-BA
> Application : domU-12-31-39-0B-80-BA
> SeqID : 488
> Data : 2013-01-10 07:44:33,021 INFO chukwa.metrics.HadoopMetrics:
> {"ScheduledReplicationBlocks":0,"BlocksTotal":131,"recordName":"FSNamesystem","CapacityTotalGB":8,"CapacityUsedGB":0,"ExcessBlocks":0,"MissingBlocks":0,"PendingReplicationBlocks":0,"timestamp":1357803873021,"FilesTotal":143,"CapacityRemainingGB":6,"UnderReplicatedBlocks":0,"context":"dfs","hostName":"domU-12-31-39-0B-80-BA","TotalLoad":1,"PendingDeletionBlocks":0,"contextName":"dfs","CorruptBlocks":0,"BlockCapacity":2097152}

Thanks,
*Shouvanik*