Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Proposed Changes To New Producer Public API


+
Jay Kreps 2014-01-31, 23:05
+
Jun Rao 2014-02-03, 18:46
+
Neha Narkhede 2014-02-03, 19:18
+
Jay Kreps 2014-02-03, 19:46
+
Joel Koshy 2014-02-03, 22:00
+
Joel Koshy 2014-02-01, 08:27
+
Joel Koshy 2014-02-01, 17:06
+
Neha Narkhede 2014-02-01, 22:53
+
Jay Kreps 2014-02-02, 17:42
+
Neha Narkhede 2014-02-02, 18:04
Copy link to this message
-
Re: Proposed Changes To New Producer Public API
About serialization, I am wondering if most people would try to have a
customized partioning mainly due to logic motivations, such that they want
some messages to go to the same partition; if that is true, they do not
really care about the actually partition id but all they would do is
specify the partition key to be the same. As for performance motivations
such as sending to the least loaded broker or broker with an existing
connection already, I am not sure if it should really be exposed to users
to consider such optimizations.
On Sun, Feb 2, 2014 at 10:04 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

>  It *is* expected that the user call this method on
> every single call and because we no longer control the partitioner
> interface there will be no way to control this.
>
> Make sense. This will ensure that new partitions are detected as defined by
> topic.metadata.refresh.interval.ms.
>
> There are a quite a few things that are inconsistent between the server and
> client. For example, java vs scala.
> Regardless, if we change the client to 'org.apache.kafka', the server
> packaging would obviously have to change
> as well to point to 'org.apache.kafka'. This feedback, to remain consistent
> with other Apache projects, was punted
> on when we first entered Apache, for no particular reason.
>
>
> On Sun, Feb 2, 2014 at 9:42 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
> > Hey Neha,
> >
> > Basically partitionsForTopic will invoke
> > Metadata.fetch(topic).partitionsFor. So it will block on the first
> request
> > for a given topic while metadata is loaded. Each subsequent request will
> > return whatever metadata is present, the logic for refreshing metadata
> will
> > remain exactly as it is. It *is* expected that the user call this method
> on
> > every single call and because we no longer control the partitioner
> > interface there will be no way to control this.
> >
> > Overall having looked at this more I think I am pretty neutral on the
> > serialization/partitioning thing. We do give up a lot by removing those
> > interfaces...
> >
> > Regardless of whether you prefer org.apache or not, the server code
> doesn't
> > have that package. So we will be inconsistent with our own code, which
> > would be extremely weird. For example the maven co-ords would be
> > different...
> >
> > -Jay
> >
> >
> >
> > On Sat, Feb 1, 2014 at 2:52 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > 1. Change send to use java.util.concurrent.Future in send():
> > >   Future<RecordPosition> send(ProducerRecord record, Callback callback)
> > > The cancel method will always return false and not do anything.
> Callback
> > > will then be changed to
> > >   interface Callback {
> > >     public void onCompletion(RecordPosition position, Exception
> > exception);
> > >   }
> > > so that the exception is available there too and will be null if no
> error
> > > occurred. I'm not planning on changing the name callback because I
> > haven't
> > > thought of a better one.
> > >
> > > +1
> > >
> > > 2. We will change the way serialization works to proposal 1A in the
> > > previous discussion. That is the Partitioner and Serializer interfaces
> > will
> > > disappear. ProducerRecord will change to:
> > >   class ProducerRecord {
> > >     public byte[] key() {...}
> > >     public byte[] value() {...}
> > >     public Integer partition() {...} // can be null
> > >   }
> > > In order to allow correctly choosing a partition the Producer interface
> > > will include a new method:
> > >   List<PartitionInfo> partitionsForTopic(String topic);
> > > PartitionInfo will be changed to include the actual Node objects not
> just
> > > the Node ids.
> > >
> > > Mostly agree with this but wanted to confirm my understanding of
> > > partitionsForTopic.
> > > So, is that something a user has to remember to invoke in order to
> detect
> > > newly
> > > added partitions to a topic? Will each invocation of partitionsForTopic
> > > lead to a
> > > TopicMetadataRequest RPC?
 
+
Jay Kreps 2014-02-02, 17:35
+
Jay Kreps 2014-02-03, 06:02
+
Jay Kreps 2014-02-03, 18:42