Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa, mail # dev - Cluster-specific Adaptors


Copy link to this message
-
Cluster-specific Adaptors
Bill Graham 2010-09-20, 21:58
Hi,

In CHUKWA-515 we discussed the possibility being able to add an
adaptor bound to a given cluster:

https://issues.apache.org/jira/browse/CHUKWA-515?focusedCommentId=12905811&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12905811

I can actually see this being useful, especially now that it's easier
to add/remove agents with the Adaptor REST API. Looking into the code
it doesn't seem like it would be that hard to do, but I want to make
sure I'm not overlooking anything.

It seems like we could support this with a few small changes:

- Add the concept of tags to the Adaptor interface.
- AbstractAdator would support a getTags method which would return the
union of tags set on the Adaptor and the default tags on the
DataFactory.
- Internal tag implementations on each would change to store tags in
maps, instead of concat'ed strings. This would allow for a "last in
wins" type of functionality so tags could be overriden. This assumes
of course that there should never be more than one of the same tag key
value, which I _assume_ is the case.
- The ChunkImpl constructor will call getTags on the agent, instead of
getDefaultTags the data factory.

The trickiest part as I see it is figuring out how to change the add
adaptor string syntax in ChukwaAgetn.processAddCommandE in a way that
both makes sense and doesn't break things. In it's current form it
doesn't have room for easy expansion except at the end of the line:

add [name =] <adaptor_class_name> <datatype> <adaptor specific params>
<initial offset>

Any thoughts or suggestions? There's also a potential gotcha with all
the impls of Adaptor.parseArgs either breaking or needing to
change....

thanks,
Bill