During the implementation of a custom Encoder/Decoder we noticed that
should the Decoder throw an exception, the KafkaStream that it is in use
becomes invalid.

Searching the mailing list indicates that the only way to recover from an
invalid stream is to shutdown the whole high-level consumer and restart.

Is there a better way to recover from this?

Thanks for the responsiveness on the mailing list.
- Bob

Wow, this seems a bit restrictive.  Can you point me to the discussion on
this?  What exactly is an "invalid stream", how can you test for this?

Jason

On Fri, Nov 9, 2012 at 8:17 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:

> During the implementation of a custom Encoder/Decoder we noticed that
> should the Decoder throw an exception, the KafkaStream that it is in use
> becomes invalid.
>
> Searching the mailing list indicates that the only way to recover from an
> invalid stream is to shutdown the whole high-level consumer and restart.
>
> Is there a better way to recover from this?
>
> Thanks for the responsiveness on the mailing list.
> - Bob
>

Here's the post I saw:
http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201201.mbox/%3CCAOG_4QbzXPJujmOjLuiHOspLfCkA+-gQP2CkN+[EMAIL PROTECTED]%3E
On Fri, Nov 9, 2012 at 10:22 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Wow, this seems a bit restrictive.  Can you point me to the discussion on
> this?  What exactly is an "invalid stream", how can you test for this?
>
> Jason
>
>
> On Fri, Nov 9, 2012 at 8:17 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
>
>> During the implementation of a custom Encoder/Decoder we noticed that
>> should the Decoder throw an exception, the KafkaStream that it is in use
>> becomes invalid.
>>
>> Searching the mailing list indicates that the only way to recover from an
>> invalid stream is to shutdown the whole high-level consumer and restart.
>>
>> Is there a better way to recover from this?
>>
>> Thanks for the responsiveness on the mailing list.
>> - Bob
>>
>
>

Currently, the code in Kafka doesn't expect the decoder to throw
exceptions. Could you write your decoder to return null on exceptions?

Thanks,

Jun

On Fri, Nov 9, 2012 at 8:17 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:

> During the implementation of a custom Encoder/Decoder we noticed that
> should the Decoder throw an exception, the KafkaStream that it is in use
> becomes invalid.
>
> Searching the mailing list indicates that the only way to recover from an
> invalid stream is to shutdown the whole high-level consumer and restart.
>
> Is there a better way to recover from this?
>
> Thanks for the responsiveness on the mailing list.
> - Bob
>

Handling the exceptions in the Decoder is not a problem.

It was just a larger question about what to do in the event the stream gets
into a failed state.

Thanks
- Bob
On Sun, Nov 11, 2012 at 11:01 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Currently, the code in Kafka doesn't expect the decoder to throw
> exceptions. Could you write your decoder to return null on exceptions?
>
> Thanks,
>
> Jun
>
>  On Fri, Nov 9, 2012 at 8:17 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
>
>> During the implementation of a custom Encoder/Decoder we noticed that
>> should the Decoder throw an exception, the KafkaStream that it is in use
>> becomes invalid.
>>
>> Searching the mailing list indicates that the only way to recover from an
>> invalid stream is to shutdown the whole high-level consumer and restart.
>>
>> Is there a better way to recover from this?
>>
>> Thanks for the responsiveness on the mailing list.
>> - Bob
>>
>
>

Bob/Jason,

This is a bug. Ideally, we should allow restarting the consumer
streams when they encounter an error.
Can you file a JIRA ?

Thanks,
Neha

On Mon, Nov 12, 2012 at 6:31 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
> Handling the exceptions in the Decoder is not a problem.
>
> It was just a larger question about what to do in the event the stream gets
> into a failed state.
>
> Thanks
> - Bob
>
>
> On Sun, Nov 11, 2012 at 11:01 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> Currently, the code in Kafka doesn't expect the decoder to throw
>> exceptions. Could you write your decoder to return null on exceptions?
>>
>> Thanks,
>>
>> Jun
>>
>>  On Fri, Nov 9, 2012 at 8:17 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
>>
>>> During the implementation of a custom Encoder/Decoder we noticed that
>>> should the Decoder throw an exception, the KafkaStream that it is in use
>>> becomes invalid.
>>>
>>> Searching the mailing list indicates that the only way to recover from an
>>> invalid stream is to shutdown the whole high-level consumer and restart.
>>>
>>> Is there a better way to recover from this?
>>>
>>> Thanks for the responsiveness on the mailing list.
>>> - Bob
>>>
>>
>>

Neha,

If I understand correctly, are you saying that it should be possible to
re-create KafkaStreams, without having to create a new ConsumerConnector?

Or are you saying that we should never get an exception when iterating over
a KafkaStream (e.g. the underlying implementation should just either
cleanly shutdown the stream, or should silently repair the stream and
continue on?).

I thought I'd clarify this before filing a Jira.....

Jason
On Tue, Nov 13, 2012 at 9:27 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Bob/Jason,
>
> This is a bug. Ideally, we should allow restarting the consumer
> streams when they encounter an error.
> Can you file a JIRA ?
>
> Thanks,
> Neha
>
> On Mon, Nov 12, 2012 at 6:31 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
> > Handling the exceptions in the Decoder is not a problem.
> >
> > It was just a larger question about what to do in the event the stream
> gets
> > into a failed state.
> >
> > Thanks
> > - Bob
> >
> >
> > On Sun, Nov 11, 2012 at 11:01 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> Currently, the code in Kafka doesn't expect the decoder to throw
> >> exceptions. Could you write your decoder to return null on exceptions?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>  On Fri, Nov 9, 2012 at 8:17 AM, Bob Cotton <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> During the implementation of a custom Encoder/Decoder we noticed that
> >>> should the Decoder throw an exception, the KafkaStream that it is in
> use
> >>> becomes invalid.
> >>>
> >>> Searching the mailing list indicates that the only way to recover from
> an
> >>> invalid stream is to shutdown the whole high-level consumer and
> restart.
> >>>
> >>> Is there a better way to recover from this?
> >>>
> >>> Thanks for the responsiveness on the mailing list.
> >>> - Bob
> >>>
> >>
> >>
>

I meant the former.

Thanks,
Neha

On Tue, Nov 13, 2012 at 10:03 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> Neha,
>
> If I understand correctly, are you saying that it should be possible to
> re-create KafkaStreams, without having to create a new ConsumerConnector?
>
> Or are you saying that we should never get an exception when iterating over
> a KafkaStream (e.g. the underlying implementation should just either
> cleanly shutdown the stream, or should silently repair the stream and
> continue on?).
>
> I thought I'd clarify this before filing a Jira.....
>
> Jason
>
>
> On Tue, Nov 13, 2012 at 9:27 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> Bob/Jason,
>>
>> This is a bug. Ideally, we should allow restarting the consumer
>> streams when they encounter an error.
>> Can you file a JIRA ?
>>
>> Thanks,
>> Neha
>>
>> On Mon, Nov 12, 2012 at 6:31 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
>> > Handling the exceptions in the Decoder is not a problem.
>> >
>> > It was just a larger question about what to do in the event the stream
>> gets
>> > into a failed state.
>> >
>> > Thanks
>> > - Bob
>> >
>> >
>> > On Sun, Nov 11, 2012 at 11:01 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> >
>> >> Currently, the code in Kafka doesn't expect the decoder to throw
>> >> exceptions. Could you write your decoder to return null on exceptions?
>> >>
>> >> Thanks,
>> >>
>> >> Jun
>> >>
>> >>  On Fri, Nov 9, 2012 at 8:17 AM, Bob Cotton <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >>> During the implementation of a custom Encoder/Decoder we noticed that
>> >>> should the Decoder throw an exception, the KafkaStream that it is in
>> use
>> >>> becomes invalid.
>> >>>
>> >>> Searching the mailing list indicates that the only way to recover from
>> an
>> >>> invalid stream is to shutdown the whole high-level consumer and
>> restart.
>> >>>
>> >>> Is there a better way to recover from this?
>> >>>
>> >>> Thanks for the responsiveness on the mailing list.
>> >>> - Bob
>> >>>
>> >>
>> >>
>>