Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa, mail # user - How to check whether chukwa collecting logs
Shouvanik Haldar
2013-01-09, 10:54
Shouvanik Haldar
2013-01-10, 07:11
Eric Yang
2013-01-10, 06:25
Shouvanik Haldar
2013-01-10, 07:01
shreyas subramanya
2013-01-10, 07:24
Shouvanik Haldar
2013-01-10, 07:34
shreyas subramanya
2013-01-10, 07:44
Shouvanik Haldar
2013-01-10, 09:42
+
Shouvanik Haldar 2013-01-09, 10:54
+
Shouvanik Haldar 2013-01-10, 07:11
+
Eric Yang 2013-01-10, 06:25
+
Shouvanik Haldar 2013-01-10, 07:01
+
shreyas subramanya 2013-01-10, 07:24
+
Shouvanik Haldar 2013-01-10, 07:34
+
shreyas subramanya 2013-01-10, 07:44
Copy link to this message
-
Re: How to check whether chukwa collecting logs
Shouvanik Haldar 2013-01-10, 08:06
Thanks.

I want to run Map Reduce Job on the .done files in hdfs. I have used the
following command

 bin/chukwa dumpArchive
hdfs://localhost:9000/chukwa/logs/201310074423346_domU1231390B80BA_675a448513c23461dfb7fa8.done

The output shows a file which contains data in a structured way like

               ==============Cluster :  cluster="chukwa"
DataType : HadoopMetrics
Source : domU-12-31-39-0B-80-BA
Application : domU-12-31-39-0B-80-BA
SeqID : 488
Data : 2013-01-10 07:44:33,021 INFO chukwa.metrics.HadoopMetrics:
{"ScheduledReplicationBlocks":0,"BlocksTotal":131,"recordName":"FSNamesystem","CapacityTotalGB":8,"CapacityUsedGB":0,"ExcessBlocks":0,"MissingBlocks":0,"PendingReplicationBlocks":0,"timestamp":1357803873021,"FilesTotal":143,"CapacityRemainingGB":6,"UnderReplicatedBlocks":0,"context":"dfs","hostName":"domU-12-31-39-0B-80-BA","TotalLoad":1,"PendingDeletionBlocks":0,"contextName":"dfs","CorruptBlocks":0,"BlockCapacity":2097152}
TimePartition: 1357801200000
DataType: HadoopMetrics
StreamName:  cluster="chukwa"/domU-12-31-39-0B-80-BA/domU-12-31-39-0B-80-BA
SeqId: 553
                 ==============Cluster :  cluster="chukwa"
DataType : HadoopMetrics
Source : domU-12-31-39-0B-80-BA
Application : domU-12-31-39-0B-80-BA
SeqID : 553
Data : 2013-01-10 07:44:33,022 INFO chukwa.metrics.HadoopMetrics:
{"port":"9000","getFileInfo_avg_time":0.0,"addBlock_num_ops":9,"rename_avg_time":3.0,"addBlock_avg_time":0.0,"recordName":"rpcdetailed","sendHeartbeat_num_ops":825,"rename_num_ops":8,"blockReceived_avg_time":0.0,"create_avg_time":3.0,"getFileInfo_num_ops":19,"timestamp":1357803873021,"sendHeartbeat_avg_time":0.0,"complete_avg_time":12.0,"complete_num_ops":9,"create_num_ops":9,"blockReceived_num_ops

Does it mean, that Map Reduce job has run?

Please help

Regards,
Shouvanik
On Thu, Jan 10, 2013 at 1:14 PM, shreyas subramanya
<[EMAIL PROTECTED]>wrote:

> Datatype is part of the metadata for a chunk. How or whether it is used
> depends on the ChukwaWriter that is configured in the collecotr. For
> example, in HBaseWriter, the datatype is used to figure out which processor
> class to invoke, which in-turn decides which hbase table to write the
> particular chunk data to. For a SeqFileWriter, it is just serialized in the
> record key. HTH.
>
> Regards,
> Shreyas
>
>
> On Wed, Jan 9, 2013 at 11:34 PM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> But what does Datatype signify here? Is it programmable.
>> What does
>>
>>
>> add FileTailer FooData /tmp/foo 0
>>
>> mean ? What is "FooData" datatype?
>>
>> Regards,
>>
>> Shouvanik
>>
>>
>>
>>
>> On Thu, Jan 10, 2013 at 12:54 PM, shreyas subramanya <
>> [EMAIL PROTECTED]> wrote:
>>
>>> FileTailer is not an adaptor, it is used by the file tailing adapors.
>>> Use FileTailingAdaptor or one of its subclasses
>>> - CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.
>>>
>>> The general syntax for these adaptors are
>>> *add FileTailingAdaptor <datatype> [offset-of-first-byte]
>>> <path-of-file> <offset>*
>>>
>>> The adaptor starts tailing from <offset-of-first-byte>
>>>
>>> Regards,
>>> Shreyas
>>>
>>>
>>> On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>
>>>> In the initial_adaptors, when I mention the below line
>>>> add FileTailer /home/ec2-user/test1.log 0
>>>>
>>>> I get a exception like
>>>> java.lang.ClassNotFoundException:
>>>> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>         at java.lang.Class.forName0(Native Method)
>>>>         at java.lang.Class.forName(Class.java:169)
Thanks,
*Shouvanik*
+
Shouvanik Haldar 2013-01-10, 09:42