Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # user >> Multiple agents and multiple clusters


+
high pointe 2011-07-14, 17:44
+
Ariel Rabkin 2011-07-14, 17:54
+
Bill Graham 2011-07-14, 22:47
+
highpointe 2011-07-15, 03:38
+
highpointe 2011-07-15, 03:45
+
Eric Yang 2011-07-15, 16:22
Copy link to this message
-
Re: Multiple agents and multiple clusters
I think that's not so hard to do.

My instinct would be to have a custom connector with two vanilla
senders in it. You'll just catch the exception when it propagates out
of the sender, and then periodically try re-initializing the sender.

--Ari

On Thu, Jul 14, 2011 at 8:45 PM, highpointe <[EMAIL PROTECTED]> wrote:
> Ahhhh. Good insight. I didn't think about that.
>
> Yes, I am aggregating the same data for two clusters.
>
> So now my challenge is:  If one cluster is unreachable...  How do I that cluster's sender fail without affecting the other sender or the connector.
>
> Excellent insight Ari!  Thank you!
>
> Sent from my iPhone
>
> On Jul 14, 2011, at 11:54 AM, Ariel Rabkin <[EMAIL PROTECTED]> wrote:
>
>> Just to be clear -- you want the same data to end up in each cluster?
>>
>> You'll have to do some hacking. I think it won't be so very monstrous.
>> In the Agent, there's a notion of pluggable Connectors and Senders. I
>> think you want a new Connector with two senders.
>>
>> One caveat -- you'll need to think through what happens if just one
>> cluster is down, and perhaps other partial failure scenarios too.
>>
>> --Ari
>>
>> On Thu, Jul 14, 2011 at 10:44 AM, high pointe <[EMAIL PROTECTED]> wrote:
>>> I have a situation where I need to take the one data stream, send it through
>>> two agents and have the end point residing in separate clusters.
>>> Rough explanation....
>>> Streaming Data Box ---> Chukwa Box
>>> Agent one on Chukwa Box sends feed to cluster one.
>>> Agent two on Chukwa box sends feed to cluster two.
>>> Does anyone know of a way to do this without hacking this baby to shreds?
>>>
>>> Thanks!
>>
>>
>>
>> --
>> Ari Rabkin [EMAIL PROTECTED]
>> UC Berkeley Computer Science Department
>

--
Ari Rabkin [EMAIL PROTECTED]
UC Berkeley Computer Science Department
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB