Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Sometimes get no message streams in consumer


Copy link to this message
-
Re: Sometimes get no message streams in consumer
It shouldn't be, no… it returns an ImmutableMap..  

setTopicMap(ImmutableMap.of(KafkaLogQueuer.sKafkaDataLogPathsTopic, 1));  

--  
Ian Friedman
On Wednesday, September 4, 2013 at 11:40 PM, Jun Rao wrote:

> What createMessageStreams() returns should match what's in the input. Is it
> possible for getTopicMap() to return an empty map?
>  
> Thanks,
>  
> Jun
>  
>  
> On Wed, Sep 4, 2013 at 1:43 PM, Ian Friedman <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>  
> > Hey gang, I have a strange intermittent issue with our 0.7.2 consumers
> > that is perpelxing me:
> >  
> > It seems like, sometimes very infrequently, when restarting our Consumer
> > fleet, some consumers threads just fail to start. The consumer connector
> > itself starts fine, and seems to be involved in subsequent rebalances, but
> > based on the logging I'm seeing, it looks like our one call to
> > ConsumerConnector.createMessageStreams is returning an empty map.
> >  
> > Is that something that should happen normally and should we be retrying in
> > our calling code? Our consumer code looks like this:
> >  
> > Map<String, List<KafkaStream<Message>>> topicMessageStreams =
> > fConsumerConnector.createMessageStreams( getTopicMap() );
> >  
> > for( Map.Entry<String, List<KafkaStream<Message>>> entry :
> > topicMessageStreams.entrySet() )
> > {
> > final String topicName = entry.getKey();
> > log.info (http://log.info)( "starting stream for topic: " + topicName );
> > final KafkaStream<Message> stream = entry.getValue().get(0);
> >  
> > ….
> >  
> > Are we doing anything dumb here? In the case I described, we're never
> > getting the "starting stream for topic: " message in our log output, so my
> > only guess is that createMessageStreams is returning an empty map.
> >  
> > Thanks in advance guys!
> >  
> > --
> > Ian Friedman
> >  
>  
>  
>