-Re: python/avro example producer
David Arthur 2012-12-19, 15:37
The fundamental unit of Kafka is a Message. A Message contains a few
bytes of metadata (a magic number, a crc32 checksum, some attributes)
and a payload of bytes. For the most part these details are obscured
from the end-user, so all you have to concern yourself with sending the
actual data (payload). In Java the payload is simply a byte array, in
Python it's just a string.
I'd suggest reading through the Quick Start
(http://kafka.apache.org/quickstart.html), and Design
(http://kafka.apache.org/design.html) if you're really interested in how
As for sending data with my Python producer, just check out the README
on the project page:
On 12/19/12 10:21 AM, Joseph Crotty wrote:
> What exactly does a "payload" mean? Sorry, fairly new to Kafka. Is
> there a payload method that needs to be called by the python producer?
> Thanks for any insights. Attached some sample code if you have time to
> lead us to the water! Probably something simple we are missing.
> On Wed, Dec 19, 2012 at 7:46 AM, David Arthur <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
> Do you mean a Python producer that sends Avro payloads?
> There are a couple of Python clients floating around, including
> mine: https://github.com/mumrah/kafka-python
> The Avro package is in pypi
> (http://pypi.python.org/pypi/avro/1.7.3), with official docs and
> getting started with Python on the Avro project page
> Good luck!
> On 12/18/12 10:38 PM, Joseph Crotty wrote:
> Anyone have a python/avro producer that slurps up records from
> a flat file
> (i.e., mix of string and binary data) and publishes to Kafka
> they would be
> willing to share?
> Starting to think this might be a whole lot faster to do in
> Java, but maybe
> someone has a Python solution already in hand.