Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Sometimes get no message streams in consumer


Copy link to this message
-
Re: Sometimes get no message streams in consumer
Jun Rao 2013-09-06, 03:28
Could you reproduce this easily?

Thanks,

Jun
On Thu, Sep 5, 2013 at 10:52 AM, Ian Friedman <[EMAIL PROTECTED]> wrote:

> It shouldn't be, no… it returns an ImmutableMap..
>
> setTopicMap(ImmutableMap.of(KafkaLogQueuer.sKafkaDataLogPathsTopic, 1));
>
> --
> Ian Friedman
>
>
> On Wednesday, September 4, 2013 at 11:40 PM, Jun Rao wrote:
>
> > What createMessageStreams() returns should match what's in the input. Is
> it
> > possible for getTopicMap() to return an empty map?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Sep 4, 2013 at 1:43 PM, Ian Friedman <[EMAIL PROTECTED] (mailto:
> [EMAIL PROTECTED])> wrote:
> >
> > > Hey gang, I have a strange intermittent issue with our 0.7.2 consumers
> > > that is perpelxing me:
> > >
> > > It seems like, sometimes very infrequently, when restarting our
> Consumer
> > > fleet, some consumers threads just fail to start. The consumer
> connector
> > > itself starts fine, and seems to be involved in subsequent rebalances,
> but
> > > based on the logging I'm seeing, it looks like our one call to
> > > ConsumerConnector.createMessageStreams is returning an empty map.
> > >
> > > Is that something that should happen normally and should we be
> retrying in
> > > our calling code? Our consumer code looks like this:
> > >
> > > Map<String, List<KafkaStream<Message>>> topicMessageStreams =
> > > fConsumerConnector.createMessageStreams( getTopicMap() );
> > >
> > > for( Map.Entry<String, List<KafkaStream<Message>>> entry :
> > > topicMessageStreams.entrySet() )
> > > {
> > > final String topicName = entry.getKey();
> > > log.info (http://log.info)( "starting stream for topic: " + topicName
> );
> > > final KafkaStream<Message> stream = entry.getValue().get(0);
> > >
> > > ….
> > >
> > > Are we doing anything dumb here? In the case I described, we're never
> > > getting the "starting stream for topic: " message in our log output,
> so my
> > > only guess is that createMessageStreams is returning an empty map.
> > >
> > > Thanks in advance guys!
> > >
> > > --
> > > Ian Friedman
> > >
> >
> >
> >
>
>
>