Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - createMessageStreamsByFilter unexpected behaviour


Copy link to this message
-
createMessageStreamsByFilter unexpected behaviour
Michal Haris 2012-11-20, 12:44
Hi, I am seeing behaviour which I am not expecting when using topic filters.

TopicFilter sourceTopicFilter = new Whitelist("pageviews");
List<KafkaStream<Message>> streams consumer.createMessageStreamsByFilter(sourceTopicFilter, 3);

The topic has exactly 3 partitions and 3 streams are created, however only
the last one gets messages, the first two become stale.
I am not sure if the singe stream contains messages from all partitions or
When I however use the following code I get all 3 streams consuming for
each respective partition.

Map<String, List<KafkaStream<Message>>> topicStreams consumer.createMessageStreams(
    new HashMap<String, Integer>() {{
        put("pageviews", 3);
    }}
);
List<KafkaStream<Message>> streams = topicStreams.get("pageviews");

Am I missing something, is this expected ?

--
Michal Haris
Software Engineer

www.visualdna.com | t: +44 (0) 207 734 7033
+
Jun Rao 2012-11-20, 17:15
+
Michal Haris 2012-11-21, 19:04