On Mon, Aug 18, 2014 at 7:27 PM, Josh J <[EMAIL PROTECTED]> wrote:
High level producer? I'm assuming you meant to write the high level consumer, in which case it isn't possible. The link above, which has an example for reading messages off a specific partition, is for the Simple consumer, which ironically, is more complex than the high level consumer.
In short, if you have a usecase where you want to read from a specific partition, you will need to implement a simple consumer.
One tactic that might be worth exploring is to rely on the message key to facilitate this.
It would require engineering careful functions for the key which hashes to the partitions for your topic(s). It would also mean that your consumers for the topic would be evaluating the key and discarding messages that aren’t relevant.
The only other option I can think of if you are using the high-level API would be finer-grained topics.
On Aug 18, 2014, at 9:14 AM, Josh J <[EMAIL PROTECTED]> wrote:
to the partitions for your topic(s). It would also mean that your consumers for the topic would be evaluating the key and discarding messages that aren’t relevant.
would be finer-grained topics.
Yes, this works great ! My producer is bucketing the messages based on the key (the key is a timestamp and I simply mod the timestamp as a function of the number of buckets). I can then do the same function on the consumer when it reads the key. I'm essentially implementing consumer sliding window. Any suggestions or tips on where I would implement reading the message key?
Thanks, Josh On Mon, Aug 18, 2014 at 6:43 PM, Jonathan Weeks <[EMAIL PROTECTED]> wrote:
The high level API gives you access to the raw Kafka event which is MessageAndMetadata, which has two pieces: key and actual message.
-Jonathan On Aug 18, 2014, at 9:50 AM, Josh J <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext