Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> python/avro example producer

Copy link to this message
Re: python/avro example producer
The fundamental unit of Kafka is a Message. A Message contains a few
bytes of metadata (a magic number, a crc32 checksum, some attributes)
and a payload of bytes. For the most part these details are obscured
from the end-user, so all you have to concern yourself with sending the
actual data (payload). In Java the payload is simply a byte array, in
Python it's just a string.

I'd suggest reading through the Quick Start
(http://kafka.apache.org/quickstart.html), and Design
(http://kafka.apache.org/design.html) if you're really interested in how
things work.

As for sending data with my Python producer, just check out the README
on the project page:


On 12/19/12 10:21 AM, Joseph Crotty wrote:
> What exactly does a "payload" mean? Sorry, fairly new to Kafka. Is
> there a payload method that needs to be called by the python producer?
> Thanks for any insights. Attached some sample code if you have time to
> lead us to the water! Probably something simple we are missing.
> Joe
> On Wed, Dec 19, 2012 at 7:46 AM, David Arthur <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>     Do you mean a Python producer that sends Avro payloads?
>     There are a couple of Python clients floating around, including
>     mine: https://github.com/mumrah/kafka-python
>     The Avro package is in pypi
>     (http://pypi.python.org/pypi/avro/1.7.3), with official docs and
>     getting started with Python on the Avro project page
>     (http://avro.apache.org/docs/1.7.3/gettingstartedpython.html)
>     Good luck!
>     On 12/18/12 10:38 PM, Joseph Crotty wrote:
>         Anyone have a python/avro producer that slurps up records from
>         a flat file
>         (i.e., mix of string and binary data) and publishes to Kafka
>         they would be
>         willing to share?
>         Starting to think this might be a whole lot faster to do in
>         Java, but maybe
>         someone has a Python solution already in hand.
>         Joe