Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Consumer doesn't start picking existing messages from topic until new data is added


Copy link to this message
-
Re: Consumer doesn't start picking existing messages from topic until new data is added
try this, not sure if it would help though

        props.put("auto.offset.reset", "smallest");

A 17/10/2013, às 14:13, Tarang Dawer <[EMAIL PROTECTED]> escreveu:

> Hi All
>
> I am facing issues with the kafka 0.8 consumer with the consumer not
> picking existing messages in the topic for consumption.
>
>
>
> Following is a snippet from my code  : -
>
>
> public void run() {
>            ConsumerIterator<byte[],byte[]> it = stream.iterator();
>            System.out.println("************************In Run : before
> while loop");   //*1st SYSO*
>            while (it.hasNext()) {
>                byte[] message = it.next().message();
>                System.out.println("Got New Mesage , Processing .. Dump in
> hbase table....********************************");   //*2nd SYSO*
> }
>
>
> Steps: -
> 1. Cleaned  zookeeper and kafka logs.
> 2. Started both again and produced 1000 messages in the topic.
> 3. Started the consumer.  the consumer , i got log till  *1st SYSO* but no
> progress. it kept on waiting for the stream to get messages
> 4. Produced 1000 more messages in the topic, now the data consumption
> starts , but the messages which i produced earlier in the topic were not
> consumed, as the hbase table showed only 1000 messages , which increased
> further depending on the additional number of messages i produced further,
> but however, the 1st initial deficit is still there.
>
> As it seems to me, the 1st 1000 messages are being lost, although the topic
> is getting created in the kafka-log directory , with the log file showing
> the initial messages there in the logbefore i start the consumer) , but the
> consumer does not seems to be picking them up.
>
> I am using Zookeeper-3.4.5 .
>
> Kafka-Settings are default , except that i the partitions for each topic
> are set to 1.
>
> Please help me out.
>
>
> Thanks
> Tarang Dawer
 -----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iF4EAREIAAYFAlJf43oACgkQLszaTQVIP3u4cwEAs8JvWtmUTGt1YRAg/+mrYHYT
eoF1RQXzGL7GVe/nL30BAIagz0+rVAsdtHquPh7icEdWbDcBKEgtMOn9YAyQDyAK
=VoAj
-----END PGP SIGNATURE-----

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB