Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Sometimes get no message streams in consumer


Copy link to this message
-
Re: Sometimes get no message streams in consumer
It shouldn't be, no… it returns an ImmutableMap..  

setTopicMap(ImmutableMap.of(KafkaLogQueuer.sKafkaDataLogPathsTopic, 1));  

--  
Ian Friedman
On Wednesday, September 4, 2013 at 11:40 PM, Jun Rao wrote:

> What createMessageStreams() returns should match what's in the input. Is it
> possible for getTopicMap() to return an empty map?
>  
> Thanks,
>  
> Jun
>  
>  
> On Wed, Sep 4, 2013 at 1:43 PM, Ian Friedman <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>  
> > Hey gang, I have a strange intermittent issue with our 0.7.2 consumers
> > that is perpelxing me:
> >  
> > It seems like, sometimes very infrequently, when restarting our Consumer
> > fleet, some consumers threads just fail to start. The consumer connector
> > itself starts fine, and seems to be involved in subsequent rebalances, but
> > based on the logging I'm seeing, it looks like our one call to
> > ConsumerConnector.createMessageStreams is returning an empty map.
> >  
> > Is that something that should happen normally and should we be retrying in
> > our calling code? Our consumer code looks like this:
> >  
> > Map<String, List<KafkaStream<Message>>> topicMessageStreams =
> > fConsumerConnector.createMessageStreams( getTopicMap() );
> >  
> > for( Map.Entry<String, List<KafkaStream<Message>>> entry :
> > topicMessageStreams.entrySet() )
> > {
> > final String topicName = entry.getKey();
> > log.info (http://log.info)( "starting stream for topic: " + topicName );
> > final KafkaStream<Message> stream = entry.getValue().get(0);
> >  
> > ….
> >  
> > Are we doing anything dumb here? In the case I described, we're never
> > getting the "starting stream for topic: " message in our log output, so my
> > only guess is that createMessageStreams is returning an empty map.
> >  
> > Thanks in advance guys!
> >  
> > --
> > Ian Friedman
> >  
>  
>  
>  

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB