Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Strategies for auto generating broker ID


Copy link to this message
-
Re: Strategies for auto generating broker ID
Guozhang Wang 2013-10-02, 16:23
Wondering what is the reason behind decoupling the node id with its
physical host(port)? If we found that for example, node 1 is not owning any
partitions, how would we know which physical machine is this node then?

Guozhang
On Wed, Oct 2, 2013 at 9:07 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> I'm in favor of doing this if someone is willing to work on it! I agree it
> would really help with easy provisioning.
>
> I filed a bug to discuss and track:
> https://issues.apache.org/jira/browse/KAFKA-1070
>
> Some comments:
> 1. I'm not in favor of having a pluggable strategy, unless we are really
> really sure this is an area where people are going to get a lot of value by
> writing lots of plugins. I am not at all sure why you would want to retain
> the current behavior if you had a good strategy for automatically
> generating ids. Basically plugins are an evil we only want to accept when
> either we don't understand the problem or the solutions have such extreme
> tradeoffs that there is no single "good approach". Plugins cause problems
> for upgrades, testing, documentation, user understandability, code
> understandability, etc.
> 2. The node id can't be the host or port or anything tied to the physical
> machine or its location on the network because you need to be able to
> change these things. I recommend we just keep an integer.
>
> -Jay
>
>
> On Tue, Oct 1, 2013 at 7:08 AM, Aniket Bhatnagar <
> [EMAIL PROTECTED]
> > wrote:
>
> > Right. It is currently java integer. However, as per previous thread, it
> > seems possible to change it to a string. In that case, we can use
> instance
> > IDs, IP addresses, custom ID generators, etc.
> > How are you currently generating broker IDs from IP address? Chef script
> or
> > custom shell script?
> >
> >
> > On 1 October 2013 18:34, Maxime Brugidou <[EMAIL PROTECTED]>
> > wrote:
> >
> > > I think it currently is a java (signed) integer or maybe this was
> > > zookeeper?
> > > We are generating the id from IP address for now but this is not ideal
> > (and
> > > can cause integer overflow with java signed ints)
> > > On Oct 1, 2013 12:52 PM, "Aniket Bhatnagar" <
> [EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > I would like to revive an older thread around auto generating broker
> > ID.
> > > As
> > > > a AWS user, I would like Kafka to just use the instance's ID or
> > > instance's
> > > > IP or instance's internal domain (whichever is easier). This would
> > mean I
> > > > can easily clone from a AMI to launch kafka instances without having
> to
> > > > worry about setting a unique broker ID. This also alows me to setup
> > auto
> > > > scaling.
> > > >
> > > > I realize 1 size may not fit all in this case. Other strategies that
> > may
> > > > work for other cloud providers are generate the UUID and persist it
> on
> > a
> > > > disk, etc.
> > > >
> > > > What I propose is a way to define a a broker ID generation strategy
> in
> > > the
> > > > configuration file which points to a class file that is responsible
> for
> > > > generating the ID. Is this something being already worked upon?
> > > >
> > >
> >
>

--
-- Guozhang