Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka Hadoop Consumer for multiple brokers


Copy link to this message
-
Re: Kafka Hadoop Consumer for multiple brokers
The idea is that each mapper is only connecting to a single Kafka broker.
Each line in the input file specifies broker uri, topic, partition and
offset.

The hadoop consumer in contrib is probably a bit outdated. The one that
LinkedIn uses now can be found at https://github.com/linkedin/camus

Thanks,

Jun
On Tue, Jun 4, 2013 at 7:29 AM, Samir Madhavan <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I was going through the hadoop-consumer in the contrib folder. There is
> property that asks for the kafka server URI. This might sound silly but
> from looking at it, it seems to be only for a single kafka broker.
>
> What we have multiple brokers, how do we implement the hadoop-consumer for
> it?
>
> Regards,
> Samir
>