Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka producer behavior


Copy link to this message
-
Re: Kafka producer behavior
Guozhang Wang 2013-12-17, 18:00
Hello,

This issue is known as in this JIRA:

https://issues.apache.org/jira/browse/KAFKA-1067

Guozhang
On Tue, Dec 17, 2013 at 8:48 AM, Gerrit Jansen van Vuuren <
[EMAIL PROTECTED]> wrote:

> hi,
>
> I've had the same issue with the kafka producer.
>
> you need to use a different partitioner than the default one provided for
> kafka.
> I've created a round robin partitioner that works well for equally
> distributing data across partitions.
>
>
> https://github.com/gerritjvv/pseidon/blob/master/pseidon-kafka/java/pseidon/kafka/util/RoundRobinPartitioner.java
>
>
>
>
>
> On Tue, Dec 17, 2013 at 5:32 PM, Hanish Bansal <
> [EMAIL PROTECTED]> wrote:
>
> > Hi All,
> >
> > We are having kafka cluster of 2 nodes. (using 0.8.0 final release)
> > Replication Factor: 2
> > Number of partitions: 2
> >
> > I have created a topic "test-topic1" in kafka.
> >
> > When i am listing status of that topic using bin/kafka-list-topic.sh, the
> > status is:
> >
> > topic: test-topic1    partition: 0    leader: 0       replicas: 0,1
> isr:
> > 0,1
> > topic: test-topic1    partition: 1    leader: 1       replicas: 1,0
> isr:
> > 1,0
> >
> > As both partition are on two separate nodes so when we produce the data
> it
> > should be go to both nodes.
> >
> > But when i insert the data, it is going to only one node.
> >
> > For example if i insert 1000 messages then all 1000 messages will go
> either
> > node1 or node2. Data is not evenly distributed on both nodes.
> >
> > Expected: 500 messages should go to node1 and 500 messages should go to
> > node2.
> >
> > Any suggestion why i am facing this behavior?
> >
> > --
> > *Thanks & Regards*
> > *Hanish Bansal*
> >
>

--
-- Guozhang