Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Add partitions command note


+
Marc Labbe 2014-01-30, 20:26
Copy link to this message
-
Re: Add partitions command note
Jay Kreps 2014-01-30, 21:29
Yeah this is confusing.

What they are trying to say is that adding partitions doesn't move data
that is already in existing partitions. I.e. say you have 10 partitions
(0-9) each containing 1GB of data and you add an 11th partition. When this
is complete partitions 0-9 will contain the exact same data as before but
there will be an empty partition 10.

This is relevant to people who are partitioning their data by some key or
other semantic feature. If you are partitioning by primary_key %
partition_count and the partition_count changes then your data is now in
the wrong partition. That is what the warning is trying to call out.

-Jay
On Thu, Jan 30, 2014 at 12:26 PM, Marc Labbe <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am going through the tools documentations for our own ops team. While
> doing so, I noticed the following note from the add partition tool:
>
> ***Please note that this tool can only be used to add partitions when data
> for a topic does not use a key.***
>
> This message left me a bit dubious as to why it would affect my topics or
> even the producer. I don't quite follow everything that's happening in
> trunk but I saw the tool has been removed there and found the equivalent in
> kafka.admin.TopicCommand alterTopic. The message there is a bit clearer and
> leaves less place for questioning.
>
> >>> WARNING: If partitions are increased for a topic that has a key, the
> partition " +   "logic or ordering of the messages will be affected"
>
> With this, I understand that adding partitions for a topic partitioned with
> a key will potentially affect order of messages consumed because they might
> end up in different partitions. This is not like adding partitions would
> not work :-)
>
> I would gladly update the wiki to clarify this for anyone looking for the
> same answer as I was.
>
> cheers,
> marc
>

 
+
Marc Labbe 2014-01-30, 21:54