Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka Hadoop Consumer for multiple brokers


Copy link to this message
-
Re: Kafka Hadoop Consumer for multiple brokers
The idea is that each mapper is only connecting to a single Kafka broker.
Each line in the input file specifies broker uri, topic, partition and
offset.

The hadoop consumer in contrib is probably a bit outdated. The one that
LinkedIn uses now can be found at https://github.com/linkedin/camus

Thanks,

Jun
On Tue, Jun 4, 2013 at 7:29 AM, Samir Madhavan <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I was going through the hadoop-consumer in the contrib folder. There is
> property that asks for the kafka server URI. This might sound silly but
> from looking at it, it seems to be only for a single kafka broker.
>
> What we have multiple brokers, how do we implement the hadoop-consumer for
> it?
>
> Regards,
> Samir
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB