Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Re: kafka replication blog


+
Michal Haris 2013-02-08, 09:41
Copy link to this message
-
Re: kafka replication blog
That's right. If you are partitioning by key, that means you insist a
message has to go to a certain partition, whether it's available or not.
So, if a partition is not available, we will drop the message for the
partition in the async mode and consistently throw an exception to the
caller in the sync mode.

Thanks,

Jun

On Fri, Feb 8, 2013 at 1:41 AM, Michal Haris <[EMAIL PROTECTED]>wrote:

> So if the produces are partitioning by key we have to have replication if
> we dont want messages to get lost when partition goes down l right ?
> Thanks
> On Feb 8, 2013 5:12 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
>
> > We have fixed this issue in 0.8. Withreplication factor 1, if the
> producer
> > doesn't care about partitioning by key, messages will be sent to
> partitions
> > that are currently available.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Feb 7, 2013 at 3:11 PM, Michal Haris <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Same here, summary was need as we have a fairly large ecosystem of
> > multiple
> > > 0.7.2 clusters and I am planning to test upgrade to 0.8.
> > > However, one thing  creeping at the back of my mind regarding 0.8 is
> > > something i have spotted in one thread few weeks ago namely that the
> > > rebalance behaviour of producers is not as robust as in 0.7.x without
> > > replication and i remeber there was no designed solution at the time -
> > any
> > > news here ? Basically our usecase doesn't require replication but
> logical
> > > offsets and some other things introduced would solve some problems.
> > > On Feb 7, 2013 7:11 PM, "Vaibhav Puranik" <[EMAIL PROTECTED]> wrote:
> > >
> > > > Same here. Thanks a lot Jun.
> > > >
> > > > Regards,
> > > > Vaibhav
> > > >
> > > > On Thu, Feb 7, 2013 at 10:38 AM, Felix GV <[EMAIL PROTECTED]>
> wrote:
> > > >
> > > > > Thanks Jun!
> > > > >
> > > > > I hadn't been following the discussions regarding 0.8 and
> replication
> > > > for a
> > > > > little while and this was a great post to refresh my memory and get
> > up
> > > to
> > > > > speed on the current replication architecture's design.
> > > > >
> > > > > --
> > > > > Felix
> > > > >
> > > > >
> > > > > On Tue, Feb 5, 2013 at 2:21 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > I just posted the following blog on Kafka replication. This may
> > > answer
> > > > > some
> > > > > > of the questions that a few people have asked in the mailing list
> > > > before.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://engineering.linkedin.com/kafka/intra-cluster-replication-apache-kafka
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > >
> > > >
> > >
> >
>

 
+
Michal Haris 2013-02-08, 20:08
+
Michal Haris 2013-02-11, 11:02
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB