Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Invalid KafkaStreams as a result of Decoder problems


Copy link to this message
-
Re: Invalid KafkaStreams as a result of Decoder problems
I meant the former.

Thanks,
Neha

On Tue, Nov 13, 2012 at 10:03 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> Neha,
>
> If I understand correctly, are you saying that it should be possible to
> re-create KafkaStreams, without having to create a new ConsumerConnector?
>
> Or are you saying that we should never get an exception when iterating over
> a KafkaStream (e.g. the underlying implementation should just either
> cleanly shutdown the stream, or should silently repair the stream and
> continue on?).
>
> I thought I'd clarify this before filing a Jira.....
>
> Jason
>
>
> On Tue, Nov 13, 2012 at 9:27 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> Bob/Jason,
>>
>> This is a bug. Ideally, we should allow restarting the consumer
>> streams when they encounter an error.
>> Can you file a JIRA ?
>>
>> Thanks,
>> Neha
>>
>> On Mon, Nov 12, 2012 at 6:31 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
>> > Handling the exceptions in the Decoder is not a problem.
>> >
>> > It was just a larger question about what to do in the event the stream
>> gets
>> > into a failed state.
>> >
>> > Thanks
>> > - Bob
>> >
>> >
>> > On Sun, Nov 11, 2012 at 11:01 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> >
>> >> Currently, the code in Kafka doesn't expect the decoder to throw
>> >> exceptions. Could you write your decoder to return null on exceptions?
>> >>
>> >> Thanks,
>> >>
>> >> Jun
>> >>
>> >>  On Fri, Nov 9, 2012 at 8:17 AM, Bob Cotton <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >>> During the implementation of a custom Encoder/Decoder we noticed that
>> >>> should the Decoder throw an exception, the KafkaStream that it is in
>> use
>> >>> becomes invalid.
>> >>>
>> >>> Searching the mailing list indicates that the only way to recover from
>> an
>> >>> invalid stream is to shutdown the whole high-level consumer and
>> restart.
>> >>>
>> >>> Is there a better way to recover from this?
>> >>>
>> >>> Thanks for the responsiveness on the mailing list.
>> >>> - Bob
>> >>>
>> >>
>> >>
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB