Chris Hogue 2013-08-02, 15:27
Thanks for bringing up this part. Yes, in 0.8, we don't really support a
custom event handler. This is because (1) the producer send logic is a bit
more complicated since it has to issue metadata requests whenever leaders
change; (2) we are not aware of too many use cases of custom event handler.
So, we only made the serializer and partitioner customizable. We can
revisit this if there are good use cases of custom event handler. For your
use case, could you just do the batching/customized compression outside of
On Fri, Aug 2, 2013 at 8:26 AM, Chris Hogue <[EMAIL PROTECTED]> wrote:
> We're a heavy 0.7 user and are now digging into 0.8 for some new projects.
> One of the features we used in 0.7 appears to be different and not clearly
> supported in 0.8.
> We use the EventHandler plug-point in 0.7, specifically to do custom
> batching before the messages are actually sent to the broker. We were
> hoping to use this feature in 0.8, albeit for a slightly different reason.
> In testing 0.8 we found the broker had a pretty significant choke point
> with compression. I'm sure this is a known performance consideration, where
> using the built-in compression forces the broker to un-compress and
> re-compress the messages to assign offsets.
> We changed our test to batch and compress the messages before it went to
> the producer, leaving the compression.codec set to none. With this change
> we saw on the order of a 3x increase in throughput on the broker while
> still retaining compressed messages across the wire and on disk.
> This was good for the tests but in the app we are planning to use semantic
> partitioning, so pre-batching and compressing before going to the producer
> is trickier. The hope was to use the producer's batching and partition
> selection, just plugging into the event handler for the compression piece.
> While I see the DefaultEventHandler is there and potentially replaceable
> via the constructor, it appears to have taken on more functionality and
> would thus be more involved to replace.
> Combined with all produce requests across brokers now being on a single
> thread, this is appearing to be difficult to achieve.
> I can imagine why some of these changes are in place in light of the
> replication feature, but wanted to throw it out there (a) in case we missed
> something or (b) at least as a potential consideration in the client
> re-design project that is under way.
> I'd be happy to have any input or other ideas on how to accomplish this
> that we may have missed. We'll look at other ways to tackle this and follow
> the other threads about the new clients in the meantime.
Chris Hogue 2013-08-02, 16:17