Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> createMessageStreamsByFilter unexpected behaviour


Copy link to this message
-
createMessageStreamsByFilter unexpected behaviour
Hi, I am seeing behaviour which I am not expecting when using topic filters.

TopicFilter sourceTopicFilter = new Whitelist("pageviews");
List<KafkaStream<Message>> streams consumer.createMessageStreamsByFilter(sourceTopicFilter, 3);

The topic has exactly 3 partitions and 3 streams are created, however only
the last one gets messages, the first two become stale.
I am not sure if the singe stream contains messages from all partitions or
When I however use the following code I get all 3 streams consuming for
each respective partition.

Map<String, List<KafkaStream<Message>>> topicStreams consumer.createMessageStreams(
    new HashMap<String, Integer>() {{
        put("pageviews", 3);
    }}
);
List<KafkaStream<Message>> streams = topicStreams.get("pageviews");

Am I missing something, is this expected ?

--
Michal Haris
Software Engineer

www.visualdna.com | t: +44 (0) 207 734 7033
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB