I have a situation where I need to take the one data stream, send it through
two agents and have the end point residing in separate clusters.

Rough explanation....

Streaming Data Box ---> Chukwa Box

Agent one on Chukwa Box sends feed to cluster one.

Agent two on Chukwa box sends feed to cluster two.

Does anyone know of a way to do this without hacking this baby to shreds?
Thanks!

Just to be clear -- you want the same data to end up in each cluster?

You'll have to do some hacking. I think it won't be so very monstrous.
In the Agent, there's a notion of pluggable Connectors and Senders. I
think you want a new Connector with two senders.

One caveat -- you'll need to think through what happens if just one
cluster is down, and perhaps other partial failure scenarios too.

--Ari

On Thu, Jul 14, 2011 at 10:44 AM, high pointe <[EMAIL PROTECTED]> wrote:
> I have a situation where I need to take the one data stream, send it through
> two agents and have the end point residing in separate clusters.
> Rough explanation....
> Streaming Data Box ---> Chukwa Box
> Agent one on Chukwa Box sends feed to cluster one.
> Agent two on Chukwa box sends feed to cluster two.
> Does anyone know of a way to do this without hacking this baby to shreds?
>
> Thanks!

--
Ari Rabkin [EMAIL PROTECTED]
UC Berkeley Computer Science Department

What's the stream source? In some cases , like file tailing, you can just
launch two adaptors both tailing the same file.

On Thu, Jul 14, 2011 at 10:54 AM, Ariel Rabkin <[EMAIL PROTECTED]> wrote:

> Just to be clear -- you want the same data to end up in each cluster?
>
> You'll have to do some hacking. I think it won't be so very monstrous.
> In the Agent, there's a notion of pluggable Connectors and Senders. I
> think you want a new Connector with two senders.
>
> One caveat -- you'll need to think through what happens if just one
> cluster is down, and perhaps other partial failure scenarios too.
>
> --Ari
>
> On Thu, Jul 14, 2011 at 10:44 AM, high pointe <[EMAIL PROTECTED]>
> wrote:
> > I have a situation where I need to take the one data stream, send it
> through
> > two agents and have the end point residing in separate clusters.
> > Rough explanation....
> > Streaming Data Box ---> Chukwa Box
> > Agent one on Chukwa Box sends feed to cluster one.
> > Agent two on Chukwa box sends feed to cluster two.
> > Does anyone know of a way to do this without hacking this baby to shreds?
> >
> > Thanks!
>
>
>
> --
> Ari Rabkin [EMAIL PROTECTED]
> UC Berkeley Computer Science Department
>

Yes. Exactly that. Tailing the same file. Just each sending to unique destinations.

Sent from my iPhone

On Jul 14, 2011, at 4:47 PM, Bill Graham <[EMAIL PROTECTED]> wrote:

> What's the stream source? In some cases , like file tailing, you can just launch two adaptors both tailing the same file.
>
> On Thu, Jul 14, 2011 at 10:54 AM, Ariel Rabkin <[EMAIL PROTECTED]> wrote:
> Just to be clear -- you want the same data to end up in each cluster?
>
> You'll have to do some hacking. I think it won't be so very monstrous.
> In the Agent, there's a notion of pluggable Connectors and Senders. I
> think you want a new Connector with two senders.
>
> One caveat -- you'll need to think through what happens if just one
> cluster is down, and perhaps other partial failure scenarios too.
>
> --Ari
>
> On Thu, Jul 14, 2011 at 10:44 AM, high pointe <[EMAIL PROTECTED]> wrote:
> > I have a situation where I need to take the one data stream, send it through
> > two agents and have the end point residing in separate clusters.
> > Rough explanation....
> > Streaming Data Box ---> Chukwa Box
> > Agent one on Chukwa Box sends feed to cluster one.
> > Agent two on Chukwa box sends feed to cluster two.
> > Does anyone know of a way to do this without hacking this baby to shreds?
> >
> > Thanks!
>
>
>
> --
> Ari Rabkin [EMAIL PROTECTED]
> UC Berkeley Computer Science Department
>

Ahhhh. Good insight. I didn't think about that.

Yes, I am aggregating the same data for two clusters.  

So now my challenge is:  If one cluster is unreachable...  How do I that cluster's sender fail without affecting the other sender or the connector.

Excellent insight Ari!  Thank you!

Sent from my iPhone

On Jul 14, 2011, at 11:54 AM, Ariel Rabkin <[EMAIL PROTECTED]> wrote:

> Just to be clear -- you want the same data to end up in each cluster?
>
> You'll have to do some hacking. I think it won't be so very monstrous.
> In the Agent, there's a notion of pluggable Connectors and Senders. I
> think you want a new Connector with two senders.
>
> One caveat -- you'll need to think through what happens if just one
> cluster is down, and perhaps other partial failure scenarios too.
>
> --Ari
>
> On Thu, Jul 14, 2011 at 10:44 AM, high pointe <[EMAIL PROTECTED]> wrote:
>> I have a situation where I need to take the one data stream, send it through
>> two agents and have the end point residing in separate clusters.
>> Rough explanation....
>> Streaming Data Box ---> Chukwa Box
>> Agent one on Chukwa Box sends feed to cluster one.
>> Agent two on Chukwa box sends feed to cluster two.
>> Does anyone know of a way to do this without hacking this baby to shreds?
>>
>> Thanks!
>
>
>
> --
> Ari Rabkin [EMAIL PROTECTED]
> UC Berkeley Computer Science Department

The data is written locally on the source node.  Both agents will progress independently, and sending data at their own pace to the collectors.  They shouldn't affect each other.  The UTF8FileTailingAdaptor should track the timestamp of the log files and send it over in time sequence order, hence one failed does not affect others.

regards,
Eric

On Jul 14, 2011, at 8:45 PM, highpointe wrote:

> Ahhhh. Good insight. I didn't think about that.
>
> Yes, I am aggregating the same data for two clusters.  
>
> So now my challenge is:  If one cluster is unreachable...  How do I that cluster's sender fail without affecting the other sender or the connector.
>
> Excellent insight Ari!  Thank you!
>
> Sent from my iPhone
>
> On Jul 14, 2011, at 11:54 AM, Ariel Rabkin <[EMAIL PROTECTED]> wrote:
>
>> Just to be clear -- you want the same data to end up in each cluster?
>>
>> You'll have to do some hacking. I think it won't be so very monstrous.
>> In the Agent, there's a notion of pluggable Connectors and Senders. I
>> think you want a new Connector with two senders.
>>
>> One caveat -- you'll need to think through what happens if just one
>> cluster is down, and perhaps other partial failure scenarios too.
>>
>> --Ari
>>
>> On Thu, Jul 14, 2011 at 10:44 AM, high pointe <[EMAIL PROTECTED]> wrote:
>>> I have a situation where I need to take the one data stream, send it through
>>> two agents and have the end point residing in separate clusters.
>>> Rough explanation....
>>> Streaming Data Box ---> Chukwa Box
>>> Agent one on Chukwa Box sends feed to cluster one.
>>> Agent two on Chukwa box sends feed to cluster two.
>>> Does anyone know of a way to do this without hacking this baby to shreds?
>>>
>>> Thanks!
>>
>>
>>
>> --
>> Ari Rabkin [EMAIL PROTECTED]
>> UC Berkeley Computer Science Department

I think that's not so hard to do.

My instinct would be to have a custom connector with two vanilla
senders in it. You'll just catch the exception when it propagates out
of the sender, and then periodically try re-initializing the sender.

--Ari

On Thu, Jul 14, 2011 at 8:45 PM, highpointe <[EMAIL PROTECTED]> wrote:
> Ahhhh. Good insight. I didn't think about that.
>
> Yes, I am aggregating the same data for two clusters.
>
> So now my challenge is:  If one cluster is unreachable...  How do I that cluster's sender fail without affecting the other sender or the connector.
>
> Excellent insight Ari!  Thank you!
>
> Sent from my iPhone
>
> On Jul 14, 2011, at 11:54 AM, Ariel Rabkin <[EMAIL PROTECTED]> wrote:
>
>> Just to be clear -- you want the same data to end up in each cluster?
>>
>> You'll have to do some hacking. I think it won't be so very monstrous.
>> In the Agent, there's a notion of pluggable Connectors and Senders. I
>> think you want a new Connector with two senders.
>>
>> One caveat -- you'll need to think through what happens if just one
>> cluster is down, and perhaps other partial failure scenarios too.
>>
>> --Ari
>>
>> On Thu, Jul 14, 2011 at 10:44 AM, high pointe <[EMAIL PROTECTED]> wrote:
>>> I have a situation where I need to take the one data stream, send it through
>>> two agents and have the end point residing in separate clusters.
>>> Rough explanation....
>>> Streaming Data Box ---> Chukwa Box
>>> Agent one on Chukwa Box sends feed to cluster one.
>>> Agent two on Chukwa box sends feed to cluster two.
>>> Does anyone know of a way to do this without hacking this baby to shreds?
>>>
>>> Thanks!
>>
>>
>>
>> --
>> Ari Rabkin [EMAIL PROTECTED]
>> UC Berkeley Computer Science Department
>

--
Ari Rabkin [EMAIL PROTECTED]
UC Berkeley Computer Science Department