Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # dev >> Cluster-specific Adaptors


Copy link to this message
-
Cluster-specific Adaptors
Hi,

In CHUKWA-515 we discussed the possibility being able to add an
adaptor bound to a given cluster:

https://issues.apache.org/jira/browse/CHUKWA-515?focusedCommentId=12905811&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12905811

I can actually see this being useful, especially now that it's easier
to add/remove agents with the Adaptor REST API. Looking into the code
it doesn't seem like it would be that hard to do, but I want to make
sure I'm not overlooking anything.

It seems like we could support this with a few small changes:

- Add the concept of tags to the Adaptor interface.
- AbstractAdator would support a getTags method which would return the
union of tags set on the Adaptor and the default tags on the
DataFactory.
- Internal tag implementations on each would change to store tags in
maps, instead of concat'ed strings. This would allow for a "last in
wins" type of functionality so tags could be overriden. This assumes
of course that there should never be more than one of the same tag key
value, which I _assume_ is the case.
- The ChunkImpl constructor will call getTags on the agent, instead of
getDefaultTags the data factory.

The trickiest part as I see it is figuring out how to change the add
adaptor string syntax in ChukwaAgetn.processAddCommandE in a way that
both makes sense and doesn't break things. In it's current form it
doesn't have room for easy expansion except at the end of the line:

add [name =] <adaptor_class_name> <datatype> <adaptor specific params>
<initial offset>

Any thoughts or suggestions? There's also a potential gotcha with all
the impls of Adaptor.parseArgs either breaking or needing to
change....

thanks,
Bill
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB