-Re: Kafka/Hadoop consumers and producers
Vadim Keylis 2013-07-03, 08:20
What is the difference between this project and Camus? Which advantages to use for loading log entries from kafka into Hadoop ?
Sent from my iPhone
On Jul 2, 2013, at 5:01 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> We currently have a contrib package for consuming and producing messages
> from mapreduce (
> We keep running into problems (e.g. KAFKA-946) that are basically due to
> the fact that the Kafka committers don't seem to mostly be Hadoop
> developers and aren't doing a good job of maintaining this code (keeping it
> tested, improving it, documenting it, writing tutorials, getting it moved
> over to the more modern apis, getting it working with newer Hadoop
> versions, etc).
> A couple of options:
> 1. We could try to get someone in the Kafka community (either a current
> committer or not) who would adopt this as their baby (it's not much code).
> 2. We could just let Camus take over this functionality. They already have
> a more sophisticated consumer and the producer is pretty minimal.
> So are there any people who would like to adopt the current Hadoop contrib
> Conversely would it be possible to provide the same or similar
> functionality in Camus and just delete these?