Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Invalid KafkaStreams as a result of Decoder problems


+
Bob Cotton 2012-11-09, 16:17
+
Jason Rosenberg 2012-11-09, 17:22
+
Bob Cotton 2012-11-09, 17:35
+
Jun Rao 2012-11-12, 06:01
+
Bob Cotton 2012-11-12, 14:31
+
Neha Narkhede 2012-11-13, 17:27
Copy link to this message
-
Re: Invalid KafkaStreams as a result of Decoder problems
Jason Rosenberg 2012-11-13, 18:03
Neha,

If I understand correctly, are you saying that it should be possible to
re-create KafkaStreams, without having to create a new ConsumerConnector?

Or are you saying that we should never get an exception when iterating over
a KafkaStream (e.g. the underlying implementation should just either
cleanly shutdown the stream, or should silently repair the stream and
continue on?).

I thought I'd clarify this before filing a Jira.....

Jason
On Tue, Nov 13, 2012 at 9:27 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Bob/Jason,
>
> This is a bug. Ideally, we should allow restarting the consumer
> streams when they encounter an error.
> Can you file a JIRA ?
>
> Thanks,
> Neha
>
> On Mon, Nov 12, 2012 at 6:31 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
> > Handling the exceptions in the Decoder is not a problem.
> >
> > It was just a larger question about what to do in the event the stream
> gets
> > into a failed state.
> >
> > Thanks
> > - Bob
> >
> >
> > On Sun, Nov 11, 2012 at 11:01 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> Currently, the code in Kafka doesn't expect the decoder to throw
> >> exceptions. Could you write your decoder to return null on exceptions?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>  On Fri, Nov 9, 2012 at 8:17 AM, Bob Cotton <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> During the implementation of a custom Encoder/Decoder we noticed that
> >>> should the Decoder throw an exception, the KafkaStream that it is in
> use
> >>> becomes invalid.
> >>>
> >>> Searching the mailing list indicates that the only way to recover from
> an
> >>> invalid stream is to shutdown the whole high-level consumer and
> restart.
> >>>
> >>> Is there a better way to recover from this?
> >>>
> >>> Thanks for the responsiveness on the mailing list.
> >>> - Bob
> >>>
> >>
> >>
>
+
Neha Narkhede 2012-11-13, 23:48