Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka 0.8 - KeyedMessage?


+
Jason Huang 2013-01-07, 16:49
+
Jun Rao 2013-01-07, 17:04
+
Jason Huang 2013-01-07, 17:31
+
Neha Narkhede 2013-01-07, 18:52
Copy link to this message
-
Re: Kafka 0.8 - KeyedMessage?
I see.

This makes sense.

thanks Neha,

Jason

On Mon, Jan 7, 2013 at 1:52 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:
> Jason,
>
> If you specify a key for a message but do not explicitly wire in a
> partitioner, messages with the same key will still land up in the same
> partition. This is because we use a default partitioner that does a simple
> hash(key) % num_partitions.
>
> Thanks,
> Neha
>
>
> On Mon, Jan 7, 2013 at 9:30 AM, Jason Huang <[EMAIL PROTECTED]> wrote:
>
>> Jun,
>>
>> Thanks for the response. If I understand you correctly, messages with
>> the same key will not be automatically stored at the same partition
>> unless I implement a partition function to route the message based on
>> the key?
>>
>> The quick start guide for 0.7 has the following:
>> "Send a message with a partition key. Messages with the same key are
>> sent to the same partition
>> ProducerData<String, String> data = new ProducerData<String,
>> String>("test-topic", "test-key", "test-message");
>> producer.send(data);"
>>
>> So... there is no simple way to send messages to a certain default
>> partition in 0.8?
>>
>> thanks,
>>
>> Jason
>>
>>
>> On Mon, Jan 7, 2013 at 12:04 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> > Jason,
>> >
>> > In 0.8, each message can optionally have a key. The key is retained as
>> part
>> > of the message and will be stored in the broker. One can design a
>> partition
>> > function to route the message based on the key. The default partitioner
>> > ignores the key and selects a partition at random.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Mon, Jan 7, 2013 at 8:49 AM, Jason Huang <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> Hello,
>> >>
>> >> I did some search on the web but couldn't find any documentation for
>> >> 0.8 so I am trying to ask here:
>> >>
>> >> KeyedMessage is introduced in 0.8.0:
>> >> class KeyedMessage[K, V](val topic: String, val key: K, val message: V)
>> >>
>> >> Does the parameter "key" = "partition key"?
>> >>
>> >> If I build a KeyedMessage with a specific key string, then the message
>> >> will be stored to this partition, right?
>> >>
>> >> If I build a KeyedMessage without a key, then the message will be
>> >> distributed to a random partition belonging to this topic? Or will it
>> >> always use a certain default partition?
>> >>
>> >> thanks,
>> >>
>> >> Jason
>> >>
>>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB