Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Strategies for auto generating broker ID


+
Aniket Bhatnagar 2013-10-01, 10:52
+
Maxime Brugidou 2013-10-01, 13:04
+
Aniket Bhatnagar 2013-10-01, 14:08
+
Jay Kreps 2013-10-02, 16:08
+
Guozhang Wang 2013-10-02, 16:23
Copy link to this message
-
Re: Strategies for auto generating broker ID
There are scenarios in which you want a hostname to change or you want to
move the stored data off one machine onto another. This is the motivation
systems have for having a layer of indirection between the location and the
identity of the nodes.

-Jay
On Wed, Oct 2, 2013 at 9:23 AM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> Wondering what is the reason behind decoupling the node id with its
> physical host(port)? If we found that for example, node 1 is not owning any
> partitions, how would we know which physical machine is this node then?
>
> Guozhang
>
>
> On Wed, Oct 2, 2013 at 9:07 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
> > I'm in favor of doing this if someone is willing to work on it! I agree
> it
> > would really help with easy provisioning.
> >
> > I filed a bug to discuss and track:
> > https://issues.apache.org/jira/browse/KAFKA-1070
> >
> > Some comments:
> > 1. I'm not in favor of having a pluggable strategy, unless we are really
> > really sure this is an area where people are going to get a lot of value
> by
> > writing lots of plugins. I am not at all sure why you would want to
> retain
> > the current behavior if you had a good strategy for automatically
> > generating ids. Basically plugins are an evil we only want to accept when
> > either we don't understand the problem or the solutions have such extreme
> > tradeoffs that there is no single "good approach". Plugins cause problems
> > for upgrades, testing, documentation, user understandability, code
> > understandability, etc.
> > 2. The node id can't be the host or port or anything tied to the physical
> > machine or its location on the network because you need to be able to
> > change these things. I recommend we just keep an integer.
> >
> > -Jay
> >
> >
> > On Tue, Oct 1, 2013 at 7:08 AM, Aniket Bhatnagar <
> > [EMAIL PROTECTED]
> > > wrote:
> >
> > > Right. It is currently java integer. However, as per previous thread,
> it
> > > seems possible to change it to a string. In that case, we can use
> > instance
> > > IDs, IP addresses, custom ID generators, etc.
> > > How are you currently generating broker IDs from IP address? Chef
> script
> > or
> > > custom shell script?
> > >
> > >
> > > On 1 October 2013 18:34, Maxime Brugidou <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > I think it currently is a java (signed) integer or maybe this was
> > > > zookeeper?
> > > > We are generating the id from IP address for now but this is not
> ideal
> > > (and
> > > > can cause integer overflow with java signed ints)
> > > > On Oct 1, 2013 12:52 PM, "Aniket Bhatnagar" <
> > [EMAIL PROTECTED]>
> > > > wrote:
> > > >
> > > > > I would like to revive an older thread around auto generating
> broker
> > > ID.
> > > > As
> > > > > a AWS user, I would like Kafka to just use the instance's ID or
> > > > instance's
> > > > > IP or instance's internal domain (whichever is easier). This would
> > > mean I
> > > > > can easily clone from a AMI to launch kafka instances without
> having
> > to
> > > > > worry about setting a unique broker ID. This also alows me to setup
> > > auto
> > > > > scaling.
> > > > >
> > > > > I realize 1 size may not fit all in this case. Other strategies
> that
> > > may
> > > > > work for other cloud providers are generate the UUID and persist it
> > on
> > > a
> > > > > disk, etc.
> > > > >
> > > > > What I propose is a way to define a a broker ID generation strategy
> > in
> > > > the
> > > > > configuration file which points to a class file that is responsible
> > for
> > > > > generating the ID. Is this something being already worked upon?
> > > > >
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

 
+
Aniket Bhatnagar 2013-10-02, 16:50
+
Jay Kreps 2013-10-02, 16:56
+
Jason Rosenberg 2013-10-02, 18:11
+
Jason Rosenberg 2013-10-02, 18:47
+
Neha Narkhede 2013-10-03, 21:31
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB