I am trying to find out a check point , so that I can say that Chukwa is
running and collecting logs
I configured the /chukwa/etc/chukwa/initial_adaptors
with following

add FileTailer
/home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
0

.
then I started bin/chukwa agent.

Started the hadoop daemons.

then, bin/chukwa collector.

Next, I check the site
"http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true"

with following statistics

Date:1357728723103
Now:1357728765604
numberHTTPConnection in time window:17
numberchunks in time window:119
lifetimechunks:23029
and when I check hdfs I see

-rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09
/chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
-rw-r--r--   1 ec2-user supergroup     483520 2013-01-08 11:14
/chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
-rw-r--r--   1 ec2-user supergroup     483155 2013-01-08 11:19
/chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
-rw-r--r--   1 ec2-user supergroup     463708 2013-01-08 11:24
/chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
-rw-r--r--   1 ec2-user supergroup     386337 2013-01-08 11:36
/chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done
-rw-r--r--   1 ec2-user supergroup     479656 2013-01-08 11:41
/chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done
-rw-r--r--   1 ec2-user supergroup     487748 2013-01-08 11:46
/chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done
-rw-r--r--   1 ec2-user supergroup     479696 2013-01-08 11:51
/chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done
-rw-r--r--   1 ec2-user supergroup     488030 2013-01-08 11:56
/chukwa/logs/201308115135034_domU12313907193B_488afeb413c19ee05bf7ff5.done
-rw-r--r--   1 ec2-user supergroup     479754 2013-01-08 12:01
/chukwa/logs/201308115635034_domU12313907193B_488afeb413c19ee05bf7ff2.done
-rw-r--r--   1 ec2-user supergroup     488067 2013-01-08 12:06
/chukwa/logs/201308120135034_domU12313907193B_488afeb413c19ee05bf7fef.done
-rw-r--r--   1 ec2-user supergroup     479961 2013-01-08 12:11
/chukwa/logs/201308120635034_domU12313907193B_488afeb413c19ee05bf7fec.done
-rw-r--r--   1 ec2-user supergroup     487976 2013-01-08 12:16
/chukwa/logs/201308121135034_domU12313907193B_488afeb413c19ee05bf7fe9.done
-rw-r--r--   1 ec2-user supergroup     479863 2013-01-08 12:21
/chukwa/logs/201308121635034_domU12313907193B_488afeb413c19ee05bf7fe6.done
-rw-r--r--   1 ec2-user supergroup     649168 2013-01-08 12:42
/chukwa/logs/201308123755390_domU12313907193B_5ab1511313c1a2ac24f8000.done
-rw-r--r--   1 ec2-user supergroup     483972 2013-01-08 12:47
/chukwa/logs/201308124255466_domU12313907193B_5ab1511313c1a2ac24f7ffc.done
-rw-r--r--   1 ec2-user supergroup     483417 2013-01-08 12:52
/chukwa/logs/201308124755466_domU12313907193B_5ab1511313c1a2ac24f7ff8.done
-rw-r--r--   1 ec2-user supergroup     576711 2013-01-08 13:06
/chukwa/logs/201308130134839_domU12313907193B_2abeddbf13c1a406b0b8000.done
-rw-r--r--   1 ec2-user supergroup     483877 2013-01-08 13:11
/chukwa/logs/201308130634912_domU12313907193B_2abeddbf13c1a406b0b7ffc.done
-rw-r--r--   1 ec2-user supergroup     484352 2013-01-08 13:16
/chukwa/logs/201308131134912_domU12313907193B_2abeddbf13c1a406b0b7ff7.done
-rw-r--r--   1 ec2-user supergroup     486158 2013-01-08 13:21
/chukwa/logs/201308131634912_domU12313907193B_2abeddbf13c1a406b0b7ff2.done
-rw-r--r--   1 ec2-user supergroup     483925 2013-01-08 13:26
/chukwa/logs/201308132134913_domU12313907193B_2abeddbf13c1a406b0b7fed.done
-rw-r--r--   1 ec2-user supergroup     629788 2013-01-08 13:37
/chukwa/logs/201308133253308_domU12313907193B_7382697b13c1a5d14db8000.done
-rw-r--r--   1 ec2-user supergroup     485716 2013-01-08 13:42
/chukwa/logs/201308133753394_domU12313907193B_7382697b13c1a5d14db7ffb.done
-rw-r--r--   1 ec2-user supergroup     486475 2013-01-08 13:47
/chukwa/logs/201308134253394_domU12313907193B_7382697b13c1a5d14db7ff5.done
-rw-r--r--   1 ec2-user supergroup     481330 2013-01-08 13:52
/chukwa/logs/201308134753394_domU12313907193B_7382697b13c1a5d14db7fef.done
-rw-r--r--   1 ec2-user supergroup     488138 2013-01-08 13:57
/chukwa/logs/201308135253394_domU12313907193B_7382697b13c1a5d14db7fe9.done
-rw-r--r--   1 ec2-user supergroup     484294 2013-01-08 14:02
/chukwa/logs/201308135753394_domU12313907193B_7382697b13c1a5d14db7fe3.done
-rw-r--r--   1 ec2-user supergroup     480689 2013-01-08 14:07
/chukwa/logs/201308140253394_domU12313907193B_7382697b13c1a5d14db7fdd.done
-rw-r--r--   1 ec2-user supergroup     488018 2013-01-08 14:12
/chukwa/logs/201308140753395_domU12313907193B_7382697b13c1a5d14db7fd7.done
-rw-r--r--   1 ec2-user supergroup     287583 2013-01-08 14:23
/chukwa/logs/201308141253395_domU12313907193B_7382697b13c1a5d14db7fd1.done
-rw-r--r--   1 ec2-user supergroup     527781 2013-01-08 14:27
/chukwa/logs/201308142254785_domU12313907193B_3065fa8513c1a8ae1598000.done
-rw-r--r--   1 ec2-user supergroup     483844 2013-01-08 14:32
/chukwa/logs/201308142754897_domU12313907193B_3065fa8513c1a8ae1597ff9.done
-rw-r--r--   1 ec2-user supergroup     483447 2013-01-08 14:37
/chukwa/logs/201308143254897_domU12313907193B_3065fa8513c1a8ae1597ff3.done
-rw-r--r--   1 ec2-user supergroup     483915 2013-01-08 14:42
/chukwa/logs/201308143754897_domU12313907193B_3065fa8513c1a8ae1597fed.done
-rw-r--r--   1 ec2-user supergroup     483902 2013-01-08 14:47
/chukwa/logs/201308144254897_domU12313907193B_3065fa8513c1a8ae1597fe7.done
-rw-r--r--   1 ec2-user supergroup     484187 2013-01-08 14:52
/chukwa/logs/201308144754897_domU12313907193B_3065fa8513c1a8ae1597fe1.done
-rw-r--r--   1 ec2-user supergroup     295291 2013-01-08 17:04
/chukwa/logs/201308145254897_domU12313907193B_3065fa8513c1a8ae1597fdb.done
-rw-r--r--   1 ec2-user supergroup     524107 2013-01-08 17:08
/chukwa/logs/201308170310783_domU12313

Can you please help me find out why the following log is generated by the
collector

2013-01-10 07:09:23,864 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:18,numberchunks:166
2013-01-10 07:09:23,884 WARN LocalToRemoteHdfsMover CopySequenceFile -
Error during .chukwa file recovery
java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
        at
org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
        at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1988)
        at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1888)
        at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1934)
        at
org.apache.hadoop.chukwa.util.CopySequenceFile.createValidSequenceFile(CopySequenceFile.java:78)
        at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.cleanup(LocalToRemoteHdfsMover.java:186)
        at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.run(LocalToRemoteHdfsMover.java:216)
2013-01-10 07:09:23,891 INFO LocalToRemoteHdfsMover LocalToRemoteHdfsMover
- Deleted .recover file,
/tmp/chukwa/dataSink/201308112441296_domU12313907193B_216836a313c19d565ee7ff8.recover
2013-01-10 07:09:23,891 INFO LocalToRemoteHdfsMover LocalToRemoteHdfsMover
- Copying .chukwa file to valid sink file before moving,
/tmp/chukwa/dataSink/201310063103378_domU1231390B80BA_1082291f13c22c778057f42.chukwa
2013-01-10 07:09:23,899 WARN LocalToRemoteHdfsMover FSInputChecker -
Problem opening checksum file:
/tmp/chukwa/dataSink/201310063103378_domU1231390B80BA_1082291f13c22c778057f42.chukwa.
Ignoring exception: java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:134)
        at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
        at
org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1495)
        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1482)
        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
        at
org.apache.hadoop.chukwa.util.CopySequenceFile.createValidSequenceFile(CopySequenceFile.java:73)
        at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.cleanup(LocalToRemoteHdfsMover.java:186)
        at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.run(LocalToRemoteHdfsMover.java:216)

2013-01-10 07:09:23,931 WARN LocalToRemoteHdfsMover CopySequenceFile -
Error during .chukwa file recovery
java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
        at
org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
        at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1988)
        at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1888)
        at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1934)
        at
org.apache.hadoop.chukwa.util.CopySequenceFile.createValidSequenceFile(CopySequenceFile.java:78)
        at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.cleanup(LocalToRemoteHdfsMover.java:186)
        at
org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalToRemoteHdfsMover.run(LocalToRemoteHdfsMover.java:216)
2013-01-10 07:09:24,348 INFO Timer-3 LocalWriter -
stat:datacollection.writer.local.LocalWriter dataSize=29534 dataRate=984
2013-01-10 07:09:54,349 INFO Timer-3 LocalWriter -
stat:datacollection.writer.local.LocalWriter dataSize=34440 dataRate=1147
Regards,
Shouvanik
On Wed, Jan 9, 2013 at 4:24 PM, Shouvanik Haldar <[EMAIL PROTECTED]
Thanks,
*Shouvanik*

Yes, the data is written to HDFS, but you probably need to run demux job to
organize the data into the form that is suitable for you.

regards,
Eric

On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <[EMAIL PROTECTED]
> wrote:

> I am trying to find out a check point , so that I can say that Chukwa is
> running and collecting logs
>
>
> I configured the /chukwa/etc/chukwa/initial_adaptors
> with following
>
> add FileTailer
> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
> 0
>
> .
> then I started bin/chukwa agent.
>
> Started the hadoop daemons.
>
> then, bin/chukwa collector.
>
> Next, I check the site
> "http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true"
>
> with following statistics
>
> Date:1357728723103
> Now:1357728765604
> numberHTTPConnection in time window:17
> numberchunks in time window:119
> lifetimechunks:23029
>
>
> and when I check hdfs I see
>
> -rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
> -rw-r--r--   1 ec2-user supergroup     483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
>
> -rw-r--r--   1 ec2-user supergroup     483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
> -rw-r--r--   1 ec2-user supergroup     463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
>
> -rw-r--r--   1 ec2-user supergroup     386337 2013-01-08 11:36 /chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done
> -rw-r--r--   1 ec2-user supergroup     479656 2013-01-08 11:41 /chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done
>
> -rw-r--r--   1 ec2-user supergroup     487748 2013-01-08 11:46 /chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done
> -rw-r--r--   1 ec2-user supergroup     479696 2013-01-08 11:51 /chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done
>
> -rw-r--r--   1 ec2-user supergroup     488030 2013-01-08 11:56 /chukwa/logs/201308115135034_domU12313907193B_488afeb413c19ee05bf7ff5.done
> -rw-r--r--   1 ec2-user supergroup     479754 2013-01-08 12:01 /chukwa/logs/201308115635034_domU12313907193B_488afeb413c19ee05bf7ff2.done
>
> -rw-r--r--   1 ec2-user supergroup     488067 2013-01-08 12:06 /chukwa/logs/201308120135034_domU12313907193B_488afeb413c19ee05bf7fef.done
> -rw-r--r--   1 ec2-user supergroup     479961 2013-01-08 12:11 /chukwa/logs/201308120635034_domU12313907193B_488afeb413c19ee05bf7fec.done
>
> -rw-r--r--   1 ec2-user supergroup     487976 2013-01-08 12:16 /chukwa/logs/201308121135034_domU12313907193B_488afeb413c19ee05bf7fe9.done
> -rw-r--r--   1 ec2-user supergroup     479863 2013-01-08 12:21 /chukwa/logs/201308121635034_domU12313907193B_488afeb413c19ee05bf7fe6.done
>
> -rw-r--r--   1 ec2-user supergroup     649168 2013-01-08 12:42 /chukwa/logs/201308123755390_domU12313907193B_5ab1511313c1a2ac24f8000.done
> -rw-r--r--   1 ec2-user supergroup     483972 2013-01-08 12:47 /chukwa/logs/201308124255466_domU12313907193B_5ab1511313c1a2ac24f7ffc.done
>
> -rw-r--r--   1 ec2-user supergroup     483417 2013-01-08 12:52 /chukwa/logs/201308124755466_domU12313907193B_5ab1511313c1a2ac24f7ff8.done
> -rw-r--r--   1 ec2-user supergroup     576711 2013-01-08 13:06 /chukwa/logs/201308130134839_domU12313907193B_2abeddbf13c1a406b0b8000.done
>
> -rw-r--r--   1 ec2-user supergroup     483877 2013-01-08 13:11 /chukwa/logs/201308130634912_domU12313907193B_2abeddbf13c1a406b0b7ffc.done
> -rw-r--r--   1 ec2-user supergroup     484352 2013-01-08 13:16 /chukwa/logs/201308131134912_domU12313907193B_2abeddbf13c1a406b0b7ff7.done
>
> -rw-r--r--   1 ec2-user supergroup     486158 2013-01-08 13:21 /chukwa/logs/201308131634912_domU12313907193B_2abeddbf13c1a406b0b7ff2.done
> -rw-r--r--   1 ec2-user supergroup     483925 2013-01-08 13:26 /chukwa/logs/201308132134913_domU12313907193B_2abeddbf13c1a406b0b7fed.done

Hi,

In the initial_adaptors, when I mention the below line
add FileTailer /home/ec2-user/test1.log 0

I get a exception like
java.lang.ClassNotFoundException:
org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at
org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
        at
org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor of
class FileTailer

*Now, my question is: *how many arguments are needed? And what is the
purpose of each argument. Having said that,

What does datatype mean in the below example?

add FileTailer FooData /tmp/foo 0

What does "FooData" datatype mean?

I created a sample file named "test1.log". How can I make Chukwa read
from this file?

Please help.

Regards,
Shouvanik
On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:

> Yes, the data is written to HDFS, but you probably need to run demux job
> to organize the data into the form that is suitable for you.
>
> regards,
> Eric
>
>
> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> I am trying to find out a check point , so that I can say that Chukwa is
>> running and collecting logs
>>
>>
>> I configured the /chukwa/etc/chukwa/initial_adaptors
>> with following
>>
>> add FileTailer
>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
>> 0
>>
>> .
>> then I started bin/chukwa agent.
>>
>> Started the hadoop daemons.
>>
>> then, bin/chukwa collector.
>>
>> Next, I check the site
>> "http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true
>> "
>>
>> with following statistics
>>
>> Date:1357728723103
>> Now:1357728765604
>> numberHTTPConnection in time window:17
>> numberchunks in time window:119
>> lifetimechunks:23029
>>
>>
>> and when I check hdfs I see
>>
>> -rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
>> -rw-r--r--   1 ec2-user supergroup     483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
>>
>>
>> -rw-r--r--   1 ec2-user supergroup     483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
>> -rw-r--r--   1 ec2-user supergroup     463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
>>
>>
>> -rw-r--r--   1 ec2-user supergroup     386337 2013-01-08 11:36 /chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done
>> -rw-r--r--   1 ec2-user supergroup     479656 2013-01-08 11:41 /chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done
>>
>>
>> -rw-r--r--   1 ec2-user supergroup     487748 2013-01-08 11:46 /chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done
>> -rw-r--r--   1 ec2-user supergroup     479696 2013-01-08 11:51 /chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done
Thanks,
*Shouvanik*

FileTailer is not an adaptor, it is used by the file tailing adapors.
Use FileTailingAdaptor or one of its subclasses
- CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.

The general syntax for these adaptors are
*add FileTailingAdaptor <datatype> [offset-of-first-byte] <path-of-file>
<offset>*

The adaptor starts tailing from <offset-of-first-byte>

Regards,
Shreyas
On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> In the initial_adaptors, when I mention the below line
> add FileTailer /home/ec2-user/test1.log 0
>
> I get a exception like
> java.lang.ClassNotFoundException:
> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:169)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
>         at
> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
> 2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor of
> class FileTailer
>
> *Now, my question is: *how many arguments are needed? And what is the
> purpose of each argument. Having said that,
>
> What does datatype mean in the below example?
>
> add FileTailer FooData /tmp/foo 0
>
>
> What does "FooData" datatype mean?
>
> I created a sample file named "test1.log". How can I make Chukwa read from this file?
>
> Please help.
>
> Regards,
>
>
> Shouvanik
>
>
>
>
> On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> Yes, the data is written to HDFS, but you probably need to run demux job
>> to organize the data into the form that is suitable for you.
>>
>> regards,
>> Eric
>>
>>
>> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
>> [EMAIL PROTECTED]> wrote:
>>
>>> I am trying to find out a check point , so that I can say that Chukwa is
>>> running and collecting logs
>>>
>>>
>>> I configured the /chukwa/etc/chukwa/initial_adaptors
>>> with following
>>>
>>> add FileTailer
>>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
>>> 0
>>>
>>> .
>>> then I started bin/chukwa agent.
>>>
>>> Started the hadoop daemons.
>>>
>>> then, bin/chukwa collector.
>>>
>>> Next, I check the site
>>> "
>>> http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true
>>> "
>>>
>>> with following statistics
>>>
>>> Date:1357728723103
>>> Now:1357728765604
>>> numberHTTPConnection in time window:17
>>> numberchunks in time window:119
>>> lifetimechunks:23029
>>>
>>>
>>> and when I check hdfs I see
>>>
>>> -rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
>>> -rw-r--r--   1 ec2-user supergroup     483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
>>>
>>>
>>>
>>>
>>> -rw-r--r--   1 ec2-user supergroup     483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
>>> -rw-r--r--   1 ec2-user supergroup     463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done

But what does Datatype signify here? Is it programmable.
What does

add FileTailer FooData /tmp/foo 0

mean ? What is "FooData" datatype?

Regards,
Shouvanik
On Thu, Jan 10, 2013 at 12:54 PM, shreyas subramanya
<[EMAIL PROTECTED]>wrote:

> FileTailer is not an adaptor, it is used by the file tailing adapors.
> Use FileTailingAdaptor or one of its subclasses
> - CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.
>
> The general syntax for these adaptors are
> *add FileTailingAdaptor <datatype> [offset-of-first-byte] <path-of-file>
> <offset>*
>
> The adaptor starts tailing from <offset-of-first-byte>
>
> Regards,
> Shreyas
>
>
> On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> In the initial_adaptors, when I mention the below line
>> add FileTailer /home/ec2-user/test1.log 0
>>
>> I get a exception like
>> java.lang.ClassNotFoundException:
>> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>         at java.lang.Class.forName0(Native Method)
>>         at java.lang.Class.forName(Class.java:169)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
>>         at
>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
>> 2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor of
>> class FileTailer
>>
>> *Now, my question is: *how many arguments are needed? And what is the
>> purpose of each argument. Having said that,
>>
>> What does datatype mean in the below example?
>>
>> add FileTailer FooData /tmp/foo 0
>>
>>
>>
>> What does "FooData" datatype mean?
>>
>> I created a sample file named "test1.log". How can I make Chukwa read from this file?
>>
>> Please help.
>>
>> Regards,
>>
>>
>>
>> Shouvanik
>>
>>
>>
>>
>> On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>
>>> Yes, the data is written to HDFS, but you probably need to run demux job
>>> to organize the data into the form that is suitable for you.
>>>
>>> regards,
>>> Eric
>>>
>>>
>>> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> I am trying to find out a check point , so that I can say that Chukwa
>>>> is running and collecting logs
>>>>
>>>>
>>>> I configured the /chukwa/etc/chukwa/initial_adaptors
>>>> with following
>>>>
>>>> add FileTailer
>>>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
>>>> 0
>>>>
>>>> .
>>>> then I started bin/chukwa agent.
>>>>
>>>> Started the hadoop daemons.
>>>>
>>>> then, bin/chukwa collector.
>>>>
>>>> Next, I check the site
>>>> "
>>>> http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true
>>>> "
>>>>
>>>> with following statistics
>>>>
>>>> Date:1357728723103
>>>> Now:1357728765604
>>>> numberHTTPConnection in time window:17
>>>> numberchunks in time window:119
>>>> lifetimechunks:23029
>>>>
>>>>
>>>> and when I check hdfs I see
>>>>
>>>> -rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
Thanks,
*Shouvanik*

Datatype is part of the metadata for a chunk. How or whether it is used
depends on the ChukwaWriter that is configured in the collecotr. For
example, in HBaseWriter, the datatype is used to figure out which processor
class to invoke, which in-turn decides which hbase table to write the
particular chunk data to. For a SeqFileWriter, it is just serialized in the
record key. HTH.

Regards,
Shreyas

On Wed, Jan 9, 2013 at 11:34 PM, Shouvanik Haldar <
[EMAIL PROTECTED]> wrote:

> But what does Datatype signify here? Is it programmable.
> What does
>
>
> add FileTailer FooData /tmp/foo 0
>
> mean ? What is "FooData" datatype?
>
> Regards,
>
> Shouvanik
>
>
>
>
> On Thu, Jan 10, 2013 at 12:54 PM, shreyas subramanya <
> [EMAIL PROTECTED]> wrote:
>
>> FileTailer is not an adaptor, it is used by the file tailing adapors.
>> Use FileTailingAdaptor or one of its subclasses
>> - CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.
>>
>> The general syntax for these adaptors are
>> *add FileTailingAdaptor <datatype> [offset-of-first-byte] <path-of-file>
>> <offset>*
>>
>> The adaptor starts tailing from <offset-of-first-byte>
>>
>> Regards,
>> Shreyas
>>
>>
>> On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> In the initial_adaptors, when I mention the below line
>>> add FileTailer /home/ec2-user/test1.log 0
>>>
>>> I get a exception like
>>> java.lang.ClassNotFoundException:
>>> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>         at java.lang.Class.forName0(Native Method)
>>>         at java.lang.Class.forName(Class.java:169)
>>>         at
>>> org.apache.hadoop.chukwa.datacollection.agent.AdaptorFactory.createAdaptor(AdaptorFactory.java:59)
>>>         at
>>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommandE(ChukwaAgent.java:447)
>>>         at
>>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.processAddCommand(ChukwaAgent.java:422)
>>>         at
>>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.readAdaptorsFile(ChukwaAgent.java:561)
>>>         at
>>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.<init>(ChukwaAgent.java:284)
>>>         at
>>> org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:169)
>>> 2013-01-10 06:56:12,653 WARN main ChukwaAgent - Error creating adaptor
>>> of class FileTailer
>>>
>>> *Now, my question is: *how many arguments are needed? And what is the
>>> purpose of each argument. Having said that,
>>>
>>> What does datatype mean in the below example?
>>>
>>> add FileTailer FooData /tmp/foo 0
>>>
>>>
>>>
>>>
>>>
>>> What does "FooData" datatype mean?
>>>
>>> I created a sample file named "test1.log". How can I make Chukwa read from this file?
>>>
>>> Please help.
>>>
>>> Regards,
>>>
>>>
>>>
>>>
>>>
>>> Shouvanik
>>>
>>>
>>>
>>>
>>> On Thu, Jan 10, 2013 at 11:55 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>>
>>>> Yes, the data is written to HDFS, but you probably need to run demux
>>>> job to organize the data into the form that is suitable for you.
>>>>
>>>> regards,
>>>> Eric
>>>>
>>>>
>>>> On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> I am trying to find out a check point , so that I can say that Chukwa
>>>>> is running and collecting logs
>>>>>
>>>>>
>>>>> I configured the /chukwa/etc/chukwa/initial_adaptors
>>>>> with following
>>>>>
>>>>> add FileTailer
>>>>> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log

Thanks.

I want to run Map Reduce Job on the .done files in hdfs. I have used the
following command

 bin/chukwa dumpArchive
hdfs://localhost:9000/chukwa/logs/201310074423346_domU1231390B80BA_675a448513c23461dfb7fa8.done

The output shows a file which contains data in a structured way like

               ==============Cluster :  cluster="chukwa"
DataType : HadoopMetrics
Source : domU-12-31-39-0B-80-BA
Application : domU-12-31-39-0B-80-BA
SeqID : 488
Data : 2013-01-10 07:44:33,021 INFO chukwa.metrics.HadoopMetrics:
{"ScheduledReplicationBlocks":0,"BlocksTotal":131,"recordName":"FSNamesystem","CapacityTotalGB":8,"CapacityUsedGB":0,"ExcessBlocks":0,"MissingBlocks":0,"PendingReplicationBlocks":0,"timestamp":1357803873021,"FilesTotal":143,"CapacityRemainingGB":6,"UnderReplicatedBlocks":0,"context":"dfs","hostName":"domU-12-31-39-0B-80-BA","TotalLoad":1,"PendingDeletionBlocks":0,"contextName":"dfs","CorruptBlocks":0,"BlockCapacity":2097152}
TimePartition: 1357801200000
DataType: HadoopMetrics
StreamName:  cluster="chukwa"/domU-12-31-39-0B-80-BA/domU-12-31-39-0B-80-BA
SeqId: 553
                 ==============Cluster :  cluster="chukwa"
DataType : HadoopMetrics
Source : domU-12-31-39-0B-80-BA
Application : domU-12-31-39-0B-80-BA
SeqID : 553
Data : 2013-01-10 07:44:33,022 INFO chukwa.metrics.HadoopMetrics:
{"port":"9000","getFileInfo_avg_time":0.0,"addBlock_num_ops":9,"rename_avg_time":3.0,"addBlock_avg_time":0.0,"recordName":"rpcdetailed","sendHeartbeat_num_ops":825,"rename_num_ops":8,"blockReceived_avg_time":0.0,"create_avg_time":3.0,"getFileInfo_num_ops":19,"timestamp":1357803873021,"sendHeartbeat_avg_time":0.0,"complete_avg_time":12.0,"complete_num_ops":9,"create_num_ops":9,"blockReceived_num_ops

Does it mean, that Map Reduce job has run?

Please help

Regards,
Shouvanik
On Thu, Jan 10, 2013 at 1:14 PM, shreyas subramanya
<[EMAIL PROTECTED]>wrote:

> Datatype is part of the metadata for a chunk. How or whether it is used
> depends on the ChukwaWriter that is configured in the collecotr. For
> example, in HBaseWriter, the datatype is used to figure out which processor
> class to invoke, which in-turn decides which hbase table to write the
> particular chunk data to. For a SeqFileWriter, it is just serialized in the
> record key. HTH.
>
> Regards,
> Shreyas
>
>
> On Wed, Jan 9, 2013 at 11:34 PM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> But what does Datatype signify here? Is it programmable.
>> What does
>>
>>
>> add FileTailer FooData /tmp/foo 0
>>
>> mean ? What is "FooData" datatype?
>>
>> Regards,
>>
>> Shouvanik
>>
>>
>>
>>
>> On Thu, Jan 10, 2013 at 12:54 PM, shreyas subramanya <
>> [EMAIL PROTECTED]> wrote:
>>
>>> FileTailer is not an adaptor, it is used by the file tailing adapors.
>>> Use FileTailingAdaptor or one of its subclasses
>>> - CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.
>>>
>>> The general syntax for these adaptors are
>>> *add FileTailingAdaptor <datatype> [offset-of-first-byte]
>>> <path-of-file> <offset>*
>>>
>>> The adaptor starts tailing from <offset-of-first-byte>
>>>
>>> Regards,
>>> Shreyas
>>>
>>>
>>> On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>
>>>> In the initial_adaptors, when I mention the below line
>>>> add FileTailer /home/ec2-user/test1.log 0
>>>>
>>>> I get a exception like
>>>> java.lang.ClassNotFoundException:
>>>> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>         at java.lang.Class.forName0(Native Method)
>>>>         at java.lang.Class.forName(Class.java:169)
Thanks,
*Shouvanik*

* I am not able to start HICC. After logging in with "admin"/"admin",

*
*getting the exception

2013-01-10 09:31:21,030 ERROR 193738361@qtp-1330491619-7 log - /hicc/
java.lang.NoSuchFieldError: STANDARD_IMPORTS
        at org.apache.jasper.compiler.PageInfo.<init>(PageInfo.java:101)
        at
org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:108)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:296)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:277)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:265)
        at
org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:564)
        at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:299)
        at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:315)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
        at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
        at
org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

*
*Please help.
*
On Thu, Jan 10, 2013 at 1:36 PM, Shouvanik Haldar <
[EMAIL PROTECTED]> wrote:

> Thanks.
>
> I want to run Map Reduce Job on the .done files in hdfs. I have used the
> following command
>
>  bin/chukwa dumpArchive
> hdfs://localhost:9000/chukwa/logs/201310074423346_domU1231390B80BA_675a448513c23461dfb7fa8.done
>
> The output shows a file which contains data in a structured way like
>
>                ==============> Cluster :  cluster="chukwa"
> DataType : HadoopMetrics
> Source : domU-12-31-39-0B-80-BA
> Application : domU-12-31-39-0B-80-BA
> SeqID : 488
> Data : 2013-01-10 07:44:33,021 INFO chukwa.metrics.HadoopMetrics:
> {"ScheduledReplicationBlocks":0,"BlocksTotal":131,"recordName":"FSNamesystem","CapacityTotalGB":8,"CapacityUsedGB":0,"ExcessBlocks":0,"MissingBlocks":0,"PendingReplicationBlocks":0,"timestamp":1357803873021,"FilesTotal":143,"CapacityRemainingGB":6,"UnderReplicatedBlocks":0,"context":"dfs","hostName":"domU-12-31-39-0B-80-BA","TotalLoad":1,"PendingDeletionBlocks":0,"contextName":"dfs","CorruptBlocks":0,"BlockCapacity":2097152}

Thanks,
*Shouvanik*