Hi there,

I am evaluating several messaging platforms for using them in combination with a CEP engine. This targets developing a system for some kind of distributed CEP. I already have built a prototype doing some very basic analysis using Kafka & Esper.

Now I am wondering about runtime modification possibilities of this system:

1.) From what I've read and done so far, I suppose I can just add new calculations  by registering new Consumers to new topics, is this correct?

2.) If so, how about updating existing calculations/applications? Will unregistering the old consumer and registering a new consumer to the same topic do the trick?

3.) Given questions 1&2 will be answered with yes, is there any way to prevent data loss while switching between versions?

In case I just did not find the appropriate part of the documentation, just give me a hint/link. Any help will be appreciated.

Regards,
Benjamin

 

Ben,

For 1), yes.

For 2) and 3), you can shutdown your consumer cleanly, do the code upgrade
and start the consumer again. On restart, Kafka will make sure that the
consumer picks up data from where it's left off and there should be no data
loss.

Thanks,

Jun

On Thu, Jan 3, 2013 at 7:07 AM, "Benjamin Süß" <[EMAIL PROTECTED]> wrote:

> Hi there,
>
> I am evaluating several messaging platforms for using them in combination
> with a CEP engine. This targets developing a system for some kind of
> distributed CEP. I already have built a prototype doing some very basic
> analysis using Kafka & Esper.
>
> Now I am wondering about runtime modification possibilities of this system:
>
> 1.) From what I've read and done so far, I suppose I can just add new
> calculations  by registering new Consumers to new topics, is this correct?
>
> 2.) If so, how about updating existing calculations/applications? Will
> unregistering the old consumer and registering a new consumer to the same
> topic do the trick?
>
> 3.) Given questions 1&2 will be answered with yes, is there any way to
> prevent data loss while switching between versions?
>
> In case I just did not find the appropriate part of the documentation,
> just give me a hint/link. Any help will be appreciated.
>
> Regards,
> Benjamin
>

 

Hi Jun,

thanks for your answers, they helped a lot.

Regards,
Benjamin

On 01/03/2013 05:16 PM, Jun Rao wrote:
> Ben,
>
> For 1), yes.
>
> For 2) and 3), you can shutdown your consumer cleanly, do the code upgrade
> and start the consumer again. On restart, Kafka will make sure that the
> consumer picks up data from where it's left off and there should be no data
> loss.
>
> Thanks,
>
> Jun
>
> On Thu, Jan 3, 2013 at 7:07 AM, "Benjamin Süß" <[EMAIL PROTECTED]> wrote:
>
>> Hi there,
>>
>> I am evaluating several messaging platforms for using them in combination
>> with a CEP engine. This targets developing a system for some kind of
>> distributed CEP. I already have built a prototype doing some very basic
>> analysis using Kafka & Esper.
>>
>> Now I am wondering about runtime modification possibilities of this system:
>>
>> 1.) From what I've read and done so far, I suppose I can just add new
>> calculations  by registering new Consumers to new topics, is this correct?
>>
>> 2.) If so, how about updating existing calculations/applications? Will
>> unregistering the old consumer and registering a new consumer to the same
>> topic do the trick?
>>
>> 3.) Given questions 1&2 will be answered with yes, is there any way to
>> prevent data loss while switching between versions?
>>
>> In case I just did not find the appropriate part of the documentation,
>> just give me a hint/link. Any help will be appreciated.
>>
>> Regards,
>> Benjamin
>>