Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - Random Partitioning Issue

Copy link to this message
Re: Random Partitioning Issue
Joel Koshy 2013-09-14, 12:17
> Thanks for bringing this up - it is definitely an important point to
> discuss. The underlying issue of KAFKA-1017 was uncovered to some degree by
> the fact that in our deployment we did not significantly increase the total
> number of partitions over 0.7 - i.e., in 0.7 we had say four partitions per
> broker, now we are using (say) eight partitions across the cluster. So with
> random partitioning every producer would end up connecting to nearly every
> broker (unlike 0.7 in which we would connect to only one broker within each
> reconnect interval). In a production-scale deployment that causes the high
> number of connections that KAFKA-1017 addresses.
> You are right that the fix of sticking to one partition over the metadata
> refresh interval goes against true consumer parallelism, but this would be
> the case only if there are few producers. If you have a sizable number of
> producers on average all partitions would get uniform volumes of data.
> One tweak to KAFKA-1017 that I think is reasonable would be instead of
> sticking to a random partition, stick to a random broker and send to random
> partitions within that broker. This would make the behavior closer to 0.7
> wrt number of connections and random partitioning provided the number of
> partitions per broker is high enough, which is why I mentioned the
> partition count (in our usage) in 0.7 vs 0.8 above. Thoughts?
> Joel
> On Friday, September 13, 2013, Joe Stein wrote:
> >
> > First, let me apologize for not realizing/noticing this until today.  One
> > reason I left my last company was not being paid to work on Kafka nor
> being
> able to afford any time for a while to work on it. Now in my new gig (just
> wrapped up my first week, woo hoo) while I am still not "paid to work on
> Kafka" I can afford some more time for it now and maybe in 6 months I will
> be able to hire folks to work on Kafka (with more and more time for myself
> to work on it too) while we also work on client projects (especially Kafka
> based ones).
> So, I understand about the changes that were made to fix open file handles
> and make the random pinning be timed based (with a very large default
> time).  Got all that.
> But, doesn't this completely negate what has been communicated to the
> community for a very long time and the expectation they have? I think it
> does.
> The expected functionality for random partitioning is that "This can be
> done in a round-robin fashion simply to balance load" and that the
> "producer" does it for you.
> Isn't a primary use case for partitions to paralyze consumers? If so then
> the expectation would be that all consumers would be getting in parallel
> equally in a "round robin fashion" the data that was produced for the
> topic... simply to balance load...with the producer handling it and with
> the client application not having to-do anything. This randomness occurring
> every 10 minutes can't balance load.
> If users are going to work around this anyways (as I would honestly do too)
> doing a pseudo semantic random key and essentially forcing real randomness
> to simply balance load to my consumers running in parallel would we still
> end up hitting the KAFKA-1017 problem anyways? If not then why can't we
> just give users the functionality and put back the 3 lines of code 1)
> if(key == null) 2)  random.nextInt(numPartitions) 3) else ... If we would
> bump into KAFKA-1017 by working around it then we have not really solved
> the root cause problem and removing expected functionality for a corner
> case that might have other work arounds and/or code changes to solve it
> another way or am I still not getting something?
> Also, I was looking at testRandomPartitioner in AsyncProducerTest and I
> don't see how this would ever fail, the assertion is always for partitionId
> == 0 and it should be checking that data is going to different partitions
> for a topic, right?
> Let me know, I think this is an important discussion and even if it ends up