Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Consumer doesn't start picking existing messages from topic until new data is added


Copy link to this message
-
Re: Consumer doesn't start picking existing messages from topic until new data is added
try this, not sure if it would help though

        props.put("auto.offset.reset", "smallest");

A 17/10/2013, às 14:13, Tarang Dawer <[EMAIL PROTECTED]> escreveu:

> Hi All
>
> I am facing issues with the kafka 0.8 consumer with the consumer not
> picking existing messages in the topic for consumption.
>
>
>
> Following is a snippet from my code  : -
>
>
> public void run() {
>            ConsumerIterator<byte[],byte[]> it = stream.iterator();
>            System.out.println("************************In Run : before
> while loop");   //*1st SYSO*
>            while (it.hasNext()) {
>                byte[] message = it.next().message();
>                System.out.println("Got New Mesage , Processing .. Dump in
> hbase table....********************************");   //*2nd SYSO*
> }
>
>
> Steps: -
> 1. Cleaned  zookeeper and kafka logs.
> 2. Started both again and produced 1000 messages in the topic.
> 3. Started the consumer.  the consumer , i got log till  *1st SYSO* but no
> progress. it kept on waiting for the stream to get messages
> 4. Produced 1000 more messages in the topic, now the data consumption
> starts , but the messages which i produced earlier in the topic were not
> consumed, as the hbase table showed only 1000 messages , which increased
> further depending on the additional number of messages i produced further,
> but however, the 1st initial deficit is still there.
>
> As it seems to me, the 1st 1000 messages are being lost, although the topic
> is getting created in the kafka-log directory , with the log file showing
> the initial messages there in the logbefore i start the consumer) , but the
> consumer does not seems to be picking them up.
>
> I am using Zookeeper-3.4.5 .
>
> Kafka-Settings are default , except that i the partitions for each topic
> are set to 1.
>
> Please help me out.
>
>
> Thanks
> Tarang Dawer
 -----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iF4EAREIAAYFAlJf43oACgkQLszaTQVIP3u4cwEAs8JvWtmUTGt1YRAg/+mrYHYT
eoF1RQXzGL7GVe/nL30BAIagz0+rVAsdtHquPh7icEdWbDcBKEgtMOn9YAyQDyAK
=VoAj
-----END PGP SIGNATURE-----