Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka producer behavior


Copy link to this message
-
Re: Kafka producer behavior
Hello,

This issue is known as in this JIRA:

https://issues.apache.org/jira/browse/KAFKA-1067

Guozhang
On Tue, Dec 17, 2013 at 8:48 AM, Gerrit Jansen van Vuuren <
[EMAIL PROTECTED]> wrote:

> hi,
>
> I've had the same issue with the kafka producer.
>
> you need to use a different partitioner than the default one provided for
> kafka.
> I've created a round robin partitioner that works well for equally
> distributing data across partitions.
>
>
> https://github.com/gerritjvv/pseidon/blob/master/pseidon-kafka/java/pseidon/kafka/util/RoundRobinPartitioner.java
>
>
>
>
>
> On Tue, Dec 17, 2013 at 5:32 PM, Hanish Bansal <
> [EMAIL PROTECTED]> wrote:
>
> > Hi All,
> >
> > We are having kafka cluster of 2 nodes. (using 0.8.0 final release)
> > Replication Factor: 2
> > Number of partitions: 2
> >
> > I have created a topic "test-topic1" in kafka.
> >
> > When i am listing status of that topic using bin/kafka-list-topic.sh, the
> > status is:
> >
> > topic: test-topic1    partition: 0    leader: 0       replicas: 0,1
> isr:
> > 0,1
> > topic: test-topic1    partition: 1    leader: 1       replicas: 1,0
> isr:
> > 1,0
> >
> > As both partition are on two separate nodes so when we produce the data
> it
> > should be go to both nodes.
> >
> > But when i insert the data, it is going to only one node.
> >
> > For example if i insert 1000 messages then all 1000 messages will go
> either
> > node1 or node2. Data is not evenly distributed on both nodes.
> >
> > Expected: 500 messages should go to node1 and 500 messages should go to
> > node2.
> >
> > Any suggestion why i am facing this behavior?
> >
> > --
> > *Thanks & Regards*
> > *Hanish Bansal*
> >
>

--
-- Guozhang

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB