Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> How to use metadata sent by agent to collector


Copy link to this message
-
Re: How to use metadata sent by agent to collector
No problem. Also note that you should be able to get the same fields
from the ChukwaRecord object if you have access to it, which would
probably be easier.

On Fri, Oct 1, 2010 at 10:52 AM, Stuti Awasthi
<[EMAIL PROTECTED]> wrote:
> Thanks Bill, I will try using chunk object.
>
> -----Original Message-----
> From: Bill Graham [mailto:[EMAIL PROTECTED]]
> Sent: Friday, October 01, 2010 10:34 PM
> To: Stuti Awasthi
> Cc: [EMAIL PROTECTED]
> Subject: Re: How to use metadata sent by agent to collector
>
> capp is chunk.streamName
> csource is chunk.source
> ctags is chunk.tags
>
> I believe body is chunk.data but note that a chunk can hold data from
> multiple records so you'd need to check chunk.recordOffsets to know
> how to split it up into multiple records.
>
>
> On Fri, Oct 1, 2010 at 9:32 AM, Stuti Awasthi
> <[EMAIL PROTECTED]> wrote:
>> Hi Bill,
>>
>> I wanted to access the "maps : " fileds
>> "mapFields": {
>>
>> "capp": "/home/hadoop/xend.log",
>> "csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
>> "ctags": " cluster=\"chukwa",
>>  "body" : "xyz"
>> }
>>
>> I checked that ChunkImpl class sets all these values. I am looking into the correct code? If yes, I can use relevant key value pair from here.
>>
>> Please Suggest
>> Regards
>> Stuti
>>
>> -----Original Message-----
>> From: Bill Graham [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, October 01, 2010 9:55 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: How to use metadata sent by agent to collector
>>
>> Can you be more specific regarding what you're trying to do? The
>> ServlerCollector, Demux and Processor classes all have access to the
>> Chunk object which encapsulated the fields you list, so it's not clear
>> what field access you're missing.
>>
>> On Fri, Oct 1, 2010 at 6:15 AM, Stuti Awasthi
>> <[EMAIL PROTECTED]> wrote:
>>> Hi all,
>>>
>>>
>>>
>>> I noticed that when agent sends chunks to collector it also sends the
>>> metadata along with it, but in demux.java class , we are just dealing with
>>> the record entry that comes from agent.
>>>
>>> How can I use this metadata information that is sent by agent to collector
>>> also on collector side.
>>>
>>>
>>>
>>> Example , each chunk sends the meta data given as below , how can I use
>>> other fields like Key  or capp on my collector side.
>>>
>>> {
>>>
>>>     "DataType": "logtype",
>>>
>>>     "Key":
>>> "1285916400000/iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com/1285917710741",
>>>
>>>     "Timestamp": 1285917710741,
>>>
>>>     "mapFields": {
>>>
>>>   "capp": "/home/hadoop/xend.log",
>>>
>>>   "csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
>>>
>>>   "ctags": " cluster=\"chukwa",
>>>
>>>  "body" : "xyz"
>>>
>>> }
>>>
>>>
>>>
>>> Any pointers will be helpful.
>>>
>>>
>>>
>>> Regards
>>>
>>> Stuti Awasthi
>>>
>>> DISCLAIMER ========== This e-mail may contain privileged and confidential
>>> information which is the property of Persistent Systems Ltd. It is intended
>>> only for the use of the individual or entity to which it is addressed. If
>>> you are not the intended recipient, you are not authorized to read, retain,
>>> copy, print, distribute or use this message. If you have received this
>>> communication in error, please notify the sender and delete all copies of
>>> this message. Persistent Systems Ltd. does not accept any liability for
>>> virus infected mails.
>>
>> DISCLAIMER
>> =========>> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>>
>
> DISCLAIMER
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB