Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> producer rewrite


Copy link to this message
-
Re: producer rewrite
Thanks.  This is the page correct?
https://reviews.apache.org/r/17263/diff/raw/

I was hoping I could download the code and run it locally not just view it.
On Thu, Jan 23, 2014 at 8:02 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> There is a link to the reviewboard url on the
> JIRA<https://issues.apache.org/jira/browse/KAFKA-1227>mentioned above.
> That will take you to the patch.
>
>
> On Thu, Jan 23, 2014 at 4:49 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
>
> > Sorry I'm new to o/s, how can I look at this patch?
> >
> > Would it be mirrored here? https://github.com/apache/kafka
> >
> >
> >
> > On Thu, Jan 23, 2014 at 3:56 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> > > Cool, I've uploaded a patch and rb here:
> > > https://issues.apache.org/jira/browse/KAFKA-1227
> > >
> > > -Jay
> > >
> > >
> > > On Thu, Jan 23, 2014 at 12:00 PM, Joe Stein <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > awesome! +1 for checking this in as is as you suggest
> > > >
> > > > /*******************************************
> > > >  Joe Stein
> > > >  Founder, Principal Consultant
> > > >  Big Data Open Source Security LLC
> > > >  http://www.stealth.ly
> > > >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > > ********************************************/
> > > >
> > > >
> > > > On Thu, Jan 23, 2014 at 2:37 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > This approach sounds reasonable to me. Since the new code will be
> not
> > > be
> > > > > used in the current kafka jar, we can still release 0.8.1 off trunk
> > > when
> > > > > it's ready.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Thu, Jan 23, 2014 at 10:23 AM, Jay Kreps <[EMAIL PROTECTED]>
> > > wrote:
> > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > I have been working on a rewrite of the producer as described in
> > the
> > > > wiki
> > > > > > below and discussed in a few previous threads:
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite
> > > > > >
> > > > > > My code is still has some bugs and is a bit rough in parts, but
> it
> > > > > > functions in the basic cases. I did some basic performance tests
> > over
> > > > > > localhost, and the new approach has paid off quite
> > significantly--for
> > > > > small
> > > > > > (10 byte) messages a single thread on my laptop can send over 1m
> > > > > > messages/second, and with larger messages easily maxes out the
> > > server.
> > > > > >
> > > > > > The difference between "sync" and "async" largely producer
> > > > > disappears--all
> > > > > > requests immediately return a future response which can be used
> to
> > > get
> > > > > the
> > > > > > behavior of either sync or async usage and we batch whenever the
> > > > producer
> > > > > > is under load using a "group commit"-like approach. You can
> > encourage
> > > > > > additional batching by incurring a small amount of latency (as
> > > before).
> > > > > >
> > > > > > Let's talk about how to integrate this code.
> > > > > >
> > > > > > This is a from-scratch rewrite of the producer code. As such it
> is
> > a
> > > > > pretty
> > > > > > major change. So far I have mostly been working on my own. I'd
> like
> > > to
> > > > > > start getting feedback before I get too far along--no point in my
> > > > > polishing
> > > > > > things that are going to be significantly revised in review,
> after
> > > all.
> > > > > >
> > > > > > As such here is what I would propose:
> > > > > >
> > > > > > 1. I'll put up a preliminary patch. Since this code is a
> completely
> > > > > > standalone module it will not destabilize the existing server or
> > > > existing
> > > > > > producer (in fact there is no change to those). I will avoid
> > > including
> > > > > > build support in this patch until we get the gradle stuff worked
> > out
> > > so
> > > > > as
> > > > > > to not break that patch (hopefully that moves along). Let's take
> > this
> > > > > patch
> > > > > > "as is" but with no expectation that the code is complete or that

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB