I am not back from my length vacation until tomorrow (really Wednesday but
going to get back to things tomorrow night) but anyways I just started to
look at where I left off with the Producer Request changes....

Looking at the ProducerPool I am not sure what I am thinking of doing with
the brokerID is what others were thinking should be done or could be done
differently.

At first glance it seems best to make the ProducerPool from this

 class ProducerPoolData[V](topic: String,
                            bidPid: Partition,
                            data: Seq[V]) {

into this

 class ProducerPoolData(brokerID: Int,producerRequest: ProducerRequest)

and refactoring accordingly?

not sure what is lost by doing the message serialization prior [V] moving
up in the layers to this but have to get back digging in the code how that
would look

then

class ProducerData[K, V](private val topic: String,
                         private val key: K,
                         private val data: java.util.List[V]) {

would change to

class ProducerData[K](
                         private val key: K,
                         private val producerRequest: ProducerRequest) {

etc...

at first blush it almost seems like ProducerPoolData can go away and some
additions to producerRequest to handle things... but I am tired and need
some rest from my vacation :)

let me know, will jump back into this tomorrow night and then back again on
Thursday/Friday hopefully (including tests) can get the ProducerRequest
wrapped up by this coming Monday (if not sooner we'll see what other if any
hurdles come about)

Thanks!

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
*/

Joe,

For kafka-240, let's just focus on replacing SyncProducer with the new wire
protocol. There is a separate jira kafka-253 on producer-side refactoring.
We can defer other changes that you want to make there.

Thanks,

Jun

On Mon, Jan 30, 2012 at 9:36 PM, Joe Stein <[EMAIL PROTECTED]> wrote:

> I am not back from my length vacation until tomorrow (really Wednesday but
> going to get back to things tomorrow night) but anyways I just started to
> look at where I left off with the Producer Request changes....
>
> Looking at the ProducerPool I am not sure what I am thinking of doing with
> the brokerID is what others were thinking should be done or could be done
> differently.
>
> At first glance it seems best to make the ProducerPool from this
>
>  class ProducerPoolData[V](topic: String,
>                            bidPid: Partition,
>                            data: Seq[V]) {
>
> into this
>
>  class ProducerPoolData(brokerID: Int,producerRequest: ProducerRequest)
>
> and refactoring accordingly?
>
> not sure what is lost by doing the message serialization prior [V] moving
> up in the layers to this but have to get back digging in the code how that
> would look
>
> then
>
> class ProducerData[K, V](private val topic: String,
>                         private val key: K,
>                         private val data: java.util.List[V]) {
>
> would change to
>
> class ProducerData[K](
>                         private val key: K,
>                         private val producerRequest: ProducerRequest) {
>
> etc...
>
> at first blush it almost seems like ProducerPoolData can go away and some
> additions to producerRequest to handle things... but I am tired and need
> some rest from my vacation :)
>
> let me know, will jump back into this tomorrow night and then back again on
> Thursday/Friday hopefully (including tests) can get the ProducerRequest
> wrapped up by this coming Monday (if not sooner we'll see what other if any
> hurdles come about)
>
> Thanks!
>
> /*
> Joe Stein
> http://www.linkedin.com/in/charmalloc
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> */
>

So now that kafka-240 is complete, shall we go ahead and start updating the
clients? i'll start updating the go client shortly..

Jeffrey,

Yes, we can get that started. Just be aware that there could be minor
tweaks to the wire protocol in the next few months.

Thanks,

Jun

On Wed, Mar 7, 2012 at 7:02 AM, Jeffrey Damick <[EMAIL PROTECTED]>wrote:

> So now that kafka-240 is complete, shall we go ahead and start updating the
> clients? i'll start updating the go client shortly..
>