Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Invalid KafkaStreams as a result of Decoder problems


Copy link to this message
-
Re: Invalid KafkaStreams as a result of Decoder problems
I meant the former.

Thanks,
Neha

On Tue, Nov 13, 2012 at 10:03 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> Neha,
>
> If I understand correctly, are you saying that it should be possible to
> re-create KafkaStreams, without having to create a new ConsumerConnector?
>
> Or are you saying that we should never get an exception when iterating over
> a KafkaStream (e.g. the underlying implementation should just either
> cleanly shutdown the stream, or should silently repair the stream and
> continue on?).
>
> I thought I'd clarify this before filing a Jira.....
>
> Jason
>
>
> On Tue, Nov 13, 2012 at 9:27 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> Bob/Jason,
>>
>> This is a bug. Ideally, we should allow restarting the consumer
>> streams when they encounter an error.
>> Can you file a JIRA ?
>>
>> Thanks,
>> Neha
>>
>> On Mon, Nov 12, 2012 at 6:31 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
>> > Handling the exceptions in the Decoder is not a problem.
>> >
>> > It was just a larger question about what to do in the event the stream
>> gets
>> > into a failed state.
>> >
>> > Thanks
>> > - Bob
>> >
>> >
>> > On Sun, Nov 11, 2012 at 11:01 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> >
>> >> Currently, the code in Kafka doesn't expect the decoder to throw
>> >> exceptions. Could you write your decoder to return null on exceptions?
>> >>
>> >> Thanks,
>> >>
>> >> Jun
>> >>
>> >>  On Fri, Nov 9, 2012 at 8:17 AM, Bob Cotton <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >>> During the implementation of a custom Encoder/Decoder we noticed that
>> >>> should the Decoder throw an exception, the KafkaStream that it is in
>> use
>> >>> becomes invalid.
>> >>>
>> >>> Searching the mailing list indicates that the only way to recover from
>> an
>> >>> invalid stream is to shutdown the whole high-level consumer and
>> restart.
>> >>>
>> >>> Is there a better way to recover from this?
>> >>>
>> >>> Thanks for the responsiveness on the mailing list.
>> >>> - Bob
>> >>>
>> >>
>> >>
>>