Kafka, mail # user - Re: high level consumer usage question with auto topic creation - 2013-07-30, 18:28
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: high level consumer usage question with auto topic creation
One thing is that you need to make sure the consumer starts consuming from
the beginning of the topic, otherwise by default, it will start from the
latest message in the topic, from the time it starts up.  Since the
consumer and producer are asynchronous, it's hard to assert that the
consumer is complete started up and ready to go, before starting the
consumer.

I had this same issue, when converting my tests from 0.7 to 0.8.  In 0.8,
the default for auto.offset.reset was changed to largest from smallest, and
so I saw the issue you are seeing.

So, you need to initialize the consumer with this property:

    cProps.put("auto.offset.reset", OffsetRequest.SmallestTimeString());
    ConsumerConfig consumerConfig = new ConsumerConfig(cProps);

Jason
On Tue, Jul 30, 2013 at 7:44 AM, Garrett Barton <[EMAIL PROTECTED]>wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB