Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> implications of using large number of topics....

Jason Rosenberg 2012-10-10, 22:57
Neha Narkhede 2012-10-10, 23:05
Jason Rosenberg 2012-10-10, 23:12
Jay Kreps 2012-10-10, 23:25
Taylor Gautier 2012-10-11, 03:13
Mathias Söderberg 2012-10-11, 13:43
Jun Rao 2012-10-12, 05:48
Jason Rosenberg 2012-10-12, 17:55
Jun Rao 2012-10-14, 03:37
Jason Rosenberg 2012-10-14, 06:42
Jun Rao 2012-10-15, 18:42
Copy link to this message
Re: implications of using large number of topics....

Thanks for the detailed response.  I'd be interested to know why you
thought it important to be able to support a large number of topics (as I'm
contemplating as well).  What was your value proposition for that (it seems
like you've gone to great lengths to make it work).

I'm curious, you mention in several places the concern about the threshold
for flushing data to disk.  And it seems you are saying that flushing
sooner than later is desirable.  Why is this?  I would have thought keeping
things in memory longer would be more efficient, etc.


On Wed, Oct 10, 2012 at 8:13 PM, Taylor Gautier <[EMAIL PROTECTED]> wrote:

> We used pattern #1 at Tagged.  I wouldn't recommend it unless you're really
> committed.  It took a lot of work to get it working right.
> a) Performance degraded non-linearly (read it fell off a cliff) when
> brokers were managing more than about 20k topics.  This was on a Linux RHEL
> 5.3 system with EXT3.  YMMV.
> b) Startup time is significantly longer for a broker that is restarted due
> to communication with ZK to sync up on those topics.
> c) If topics are short lived, even if Kafka expires the data segments using
> it's standard 0.7 cleaner, the directory name for the topic will still
> exist on disk and the topic is still considered "active" (in memory) in
> Kafka.  This causes problems - see a above (open file handles).
> d) Message latency is affected.  Kafka syncs messages to disk if x messages
> have buffered in memory, or y seconds have elapsed (both configurable).  If
> you have few topics and many messages (pattern #2), you will be hitting the
> x limit quite often, and get good throughput.  However, if you have many
> topics and few messages per topic (pattern #1), you will have to rely on
> the y threshold to flush to disk, and setting this too low can impact
> performance (throughput) in a significant way.  Jay already mentioned this
> as random writes.
> We had to implement a number of solutions ourselves to resolve these
> issues, namely:
> #1 No ZK.  This means that all of the automatic partitioning done by Kafka
> is not available, so we had to roll our own (luckily Tagged is pretty used
> to scaling systems so there was much in-house expertise).  The solution
> here was to implement a R/W proxy layer of machines to intercept messages
> and read/write to/from Kafka handling the sharding at the proxy layer.
>  Because most of our messages were coming from PHP and we didn't want to
> use TCP we needed a UDP/TCP bridge/proxy anyway so this wasn't a huge deal
> (also, we wanted strict ordering of messages, so we needed a shard by topic
> feature anyway (I believe this can be done in 0.7 but we started with 0.6)
> #2 Custom cleaner.  We implemented an extra cleanup task inside the Kafka
> process that could completely remove a topic from memory and disk.  For
> clients, this sometimes meant that a subscribed topic suddenly changed it's
> physical offset from some offset X to 0, but that's ok, while technically
> it probably would never happen theoretically clients should have to handle
> this case anyway because the Kafka physical message space is limited to
> 64-bits (again, unlikely to ever wrap in practice, but you never know).
>  Anyway it's pretty easy to handle this just catch the "invalid offset"
> error Kafka gives and start at 0.
> #3 Low threshold for flush.  This gave us good latency, but poor throughput
> (relatively speaking).  We had more than enough throughput, but it was
> nowhere near what Kafka can do when setup in pattern #1.
> Given that you want to manage "hundreds of thousands of topics" that may
> mean that you would need 10's of Kafka brokers which could be another
> source of problems - it's more cost, more management, and more sources of
> failure.  SSD's may help solve this problem btw, but now you are talking
> expensive machines rather than using just off the shelf cheapo servers with
> standard SATA drives.
> On Wed, Oct 10, 2012 at 4:25 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
Taylor Gautier 2012-10-11, 17:08