Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Broker to consumer compression

Pablo Barrera González 2013-04-12, 17:57
Copy link to this message
Re: Broker to consumer compression
Kafka already supports end-to-end compression which means data
transfer between brokers and consumers is compressed. There are two
supported compression codecs - GZIP and Snappy. The latter is lighter
on CPU consumption. See this blog post for comparison -


On Fri, Apr 12, 2013 at 10:56 AM, Pablo Barrera González
> Hi
> Is it possible to enable compression between the broker and the consumer?
> We are thinking in develop this feature in kafka 0.7 but first I would
> like to check if there is something out there.
> Our escenario is like this:
> - the producer is a CPU bounded machine, so we want to keep the CPU
> consumption as low as possible, so we can't enable compression here
> - the consumers can fetch data from the same data center (no
> compression needed) or from a remote data center
> - intersite bandwidth is limited so compression would be interesting
> Our approach is to add compress the connection at kafka level between
> broker and consumer, inside kafka, so the final user can read plain
> data.
> Regards
> Pablo

Pablo Barrera González 2013-04-12, 19:59
Neha Narkhede 2013-04-12, 21:32