Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Strategies for auto generating broker ID


+
Aniket Bhatnagar 2013-10-01, 10:52
+
Maxime Brugidou 2013-10-01, 13:04
+
Aniket Bhatnagar 2013-10-01, 14:08
+
Jay Kreps 2013-10-02, 16:08
+
Guozhang Wang 2013-10-02, 16:23
Copy link to this message
-
Re: Strategies for auto generating broker ID
Jay Kreps 2013-10-02, 16:53
There are scenarios in which you want a hostname to change or you want to
move the stored data off one machine onto another. This is the motivation
systems have for having a layer of indirection between the location and the
identity of the nodes.

-Jay
On Wed, Oct 2, 2013 at 9:23 AM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> Wondering what is the reason behind decoupling the node id with its
> physical host(port)? If we found that for example, node 1 is not owning any
> partitions, how would we know which physical machine is this node then?
>
> Guozhang
>
>
> On Wed, Oct 2, 2013 at 9:07 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
> > I'm in favor of doing this if someone is willing to work on it! I agree
> it
> > would really help with easy provisioning.
> >
> > I filed a bug to discuss and track:
> > https://issues.apache.org/jira/browse/KAFKA-1070
> >
> > Some comments:
> > 1. I'm not in favor of having a pluggable strategy, unless we are really
> > really sure this is an area where people are going to get a lot of value
> by
> > writing lots of plugins. I am not at all sure why you would want to
> retain
> > the current behavior if you had a good strategy for automatically
> > generating ids. Basically plugins are an evil we only want to accept when
> > either we don't understand the problem or the solutions have such extreme
> > tradeoffs that there is no single "good approach". Plugins cause problems
> > for upgrades, testing, documentation, user understandability, code
> > understandability, etc.
> > 2. The node id can't be the host or port or anything tied to the physical
> > machine or its location on the network because you need to be able to
> > change these things. I recommend we just keep an integer.
> >
> > -Jay
> >
> >
> > On Tue, Oct 1, 2013 at 7:08 AM, Aniket Bhatnagar <
> > [EMAIL PROTECTED]
> > > wrote:
> >
> > > Right. It is currently java integer. However, as per previous thread,
> it
> > > seems possible to change it to a string. In that case, we can use
> > instance
> > > IDs, IP addresses, custom ID generators, etc.
> > > How are you currently generating broker IDs from IP address? Chef
> script
> > or
> > > custom shell script?
> > >
> > >
> > > On 1 October 2013 18:34, Maxime Brugidou <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > I think it currently is a java (signed) integer or maybe this was
> > > > zookeeper?
> > > > We are generating the id from IP address for now but this is not
> ideal
> > > (and
> > > > can cause integer overflow with java signed ints)
> > > > On Oct 1, 2013 12:52 PM, "Aniket Bhatnagar" <
> > [EMAIL PROTECTED]>
> > > > wrote:
> > > >
> > > > > I would like to revive an older thread around auto generating
> broker
> > > ID.
> > > > As
> > > > > a AWS user, I would like Kafka to just use the instance's ID or
> > > > instance's
> > > > > IP or instance's internal domain (whichever is easier). This would
> > > mean I
> > > > > can easily clone from a AMI to launch kafka instances without
> having
> > to
> > > > > worry about setting a unique broker ID. This also alows me to setup
> > > auto
> > > > > scaling.
> > > > >
> > > > > I realize 1 size may not fit all in this case. Other strategies
> that
> > > may
> > > > > work for other cloud providers are generate the UUID and persist it
> > on
> > > a
> > > > > disk, etc.
> > > > >
> > > > > What I propose is a way to define a a broker ID generation strategy
> > in
> > > > the
> > > > > configuration file which points to a class file that is responsible
> > for
> > > > > generating the ID. Is this something being already worked upon?
> > > > >
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

 
+
Aniket Bhatnagar 2013-10-02, 16:50
+
Jay Kreps 2013-10-02, 16:56
+
Jason Rosenberg 2013-10-02, 18:11
+
Jason Rosenberg 2013-10-02, 18:47
+
Neha Narkhede 2013-10-03, 21:31