Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> How to check whether chukwa collecting logs


Copy link to this message
-
Re: How to check whether chukwa collecting logs
Thanks.

I want to run Map Reduce Job on the .done files in hdfs. I have used the
following command

 bin/chukwa dumpArchive
hdfs://localhost:9000/chukwa/logs/201310074423346_domU1231390B80BA_675a448513c23461dfb7fa8.done

The output shows a file which contains data in a structured way like

               ==============Cluster :  cluster="chukwa"
DataType : HadoopMetrics
Source : domU-12-31-39-0B-80-BA
Application : domU-12-31-39-0B-80-BA
SeqID : 488
Data : 2013-01-10 07:44:33,021 INFO chukwa.metrics.HadoopMetrics:
{"ScheduledReplicationBlocks":0,"BlocksTotal":131,"recordName":"FSNamesystem","CapacityTotalGB":8,"CapacityUsedGB":0,"ExcessBlocks":0,"MissingBlocks":0,"PendingReplicationBlocks":0,"timestamp":1357803873021,"FilesTotal":143,"CapacityRemainingGB":6,"UnderReplicatedBlocks":0,"context":"dfs","hostName":"domU-12-31-39-0B-80-BA","TotalLoad":1,"PendingDeletionBlocks":0,"contextName":"dfs","CorruptBlocks":0,"BlockCapacity":2097152}
TimePartition: 1357801200000
DataType: HadoopMetrics
StreamName:  cluster="chukwa"/domU-12-31-39-0B-80-BA/domU-12-31-39-0B-80-BA
SeqId: 553
                 ==============Cluster :  cluster="chukwa"
DataType : HadoopMetrics
Source : domU-12-31-39-0B-80-BA
Application : domU-12-31-39-0B-80-BA
SeqID : 553
Data : 2013-01-10 07:44:33,022 INFO chukwa.metrics.HadoopMetrics:
{"port":"9000","getFileInfo_avg_time":0.0,"addBlock_num_ops":9,"rename_avg_time":3.0,"addBlock_avg_time":0.0,"recordName":"rpcdetailed","sendHeartbeat_num_ops":825,"rename_num_ops":8,"blockReceived_avg_time":0.0,"create_avg_time":3.0,"getFileInfo_num_ops":19,"timestamp":1357803873021,"sendHeartbeat_avg_time":0.0,"complete_avg_time":12.0,"complete_num_ops":9,"create_num_ops":9,"blockReceived_num_ops

Does it mean, that Map Reduce job has run?

Please help

Regards,
Shouvanik
On Thu, Jan 10, 2013 at 1:14 PM, shreyas subramanya
<[EMAIL PROTECTED]>wrote:

> Datatype is part of the metadata for a chunk. How or whether it is used
> depends on the ChukwaWriter that is configured in the collecotr. For
> example, in HBaseWriter, the datatype is used to figure out which processor
> class to invoke, which in-turn decides which hbase table to write the
> particular chunk data to. For a SeqFileWriter, it is just serialized in the
> record key. HTH.
>
> Regards,
> Shreyas
>
>
> On Wed, Jan 9, 2013 at 11:34 PM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> But what does Datatype signify here? Is it programmable.
>> What does
>>
>>
>> add FileTailer FooData /tmp/foo 0
>>
>> mean ? What is "FooData" datatype?
>>
>> Regards,
>>
>> Shouvanik
>>
>>
>>
>>
>> On Thu, Jan 10, 2013 at 12:54 PM, shreyas subramanya <
>> [EMAIL PROTECTED]> wrote:
>>
>>> FileTailer is not an adaptor, it is used by the file tailing adapors.
>>> Use FileTailingAdaptor or one of its subclasses
>>> - CharFileTailingAdaptorUTF8 or CharFileTailingAdaptorUTF8NewLineEscaped.
>>>
>>> The general syntax for these adaptors are
>>> *add FileTailingAdaptor <datatype> [offset-of-first-byte]
>>> <path-of-file> <offset>*
>>>
>>> The adaptor starts tailing from <offset-of-first-byte>
>>>
>>> Regards,
>>> Shreyas
>>>
>>>
>>> On Wed, Jan 9, 2013 at 11:01 PM, Shouvanik Haldar <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>
>>>> In the initial_adaptors, when I mention the below line
>>>> add FileTailer /home/ec2-user/test1.log 0
>>>>
>>>> I get a exception like
>>>> java.lang.ClassNotFoundException:
>>>> org.apache.hadoop.chukwa.datacollection.adaptor.FileTailer
>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>         at java.lang.Class.forName0(Native Method)
>>>>         at java.lang.Class.forName(Class.java:169)
Thanks,
*Shouvanik*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB