Thanks for confirming, but that not behavior I observe. My consumer does not commit data to kafka. It get messages sent to kafka. Once restarted I should of gotten messages that previously received by consumer, but on contrarily I got none. Logs confirm the initial offset been as -1. What am I doing wrong?
04 Nov 2013 16:03:11,570 DEBUG meetme_Consumer_pkey_1062739249349868 kafka.consumer.PartitionTopicInfo - initial consumer offset of meetme:0: fetched offset = -1: consumed offset = -1 is -1 04 Nov 2013 16:03:11,570 DEBUG meetme_Consumer_pkey_1062739249349868 kafka.consumer.PartitionTopicInfo - initial fetch offset of meetme:0: fetched offset = -1: consumed offset = -1 is -1
04 Nov 2013 16:03:11,879 DEBUG event1_ddatahubvadim02.tag-dev.com-1383609790143-4ed618e7-leader-finder-thread kafka.network.BlockingChannel - Created socket with SO_TIMEOUT = 30000 (requested 30000), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF = 11460 (requested -1). 04 Nov 2013 16:03:11,895 DEBUG event1_ddatahubvadim02.tag-dev.com-1383609790143-4ed618e7-leader-finder-thread kafka.consumer.PartitionTopicInfo - reset fetch offset of ( meetme:0: fetched offset = 99000: consumed offset = -1 ) to 99000 04 Nov 2013 16:03:11,896 DEBUG event1_ddatahubvadim02.tag-dev.com-1383609790143-4ed618e7-leader-finder-thread kafka.consumer.PartitionTopicInfo - reset consume offset of meetme:0: fetched offset = 99000: consumed offset = 99000 to 99000 04 Nov 2013 16:03:11,897 INFO event1_ddatahubvadim02.tag-dev.com-1383609790143-4ed618e7-leader-finder-thread kafka.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1383609790333] Adding fetcher for partition [meetme,0], initOffset -1 to broker 9 with fetcherId 0 Here is my property file: zookeeper.connect=dzoo01.tag-dev.com:2181/kafka zookeeper.connectiontimeout.ms=1000000 group.id=event1 auto.commit.enable=false On Mon, Nov 4, 2013 at 3:32 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
I am using creating Consumer.createJavaConsumerConnector(kafka 0.8) for each topic/partition. Would it be safe to assume that commit offset will apply only to stream/partition managed by that connector?
Thanks, Vadim On Mon, Nov 4, 2013 at 8:43 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
That I can manage. Thanks so much. On Tue, Nov 5, 2013 at 6:46 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext