Hi -
I see there's a static broker-id in the properties files. Is this optional?
When using Zookeeper, couldn't ZK just give each broker if registered as a
SEQ node a generated ID that's guaranteed to be unique? Otherwise can this
ID be a randomly generated ID? Basically I'd like to bring up instances in
EC2 without having to worry about the state of other brokers...

Thanks!
Flo

Florian,

The broker id uniquely identifies a broker. Every time a broker is
restarted, we want the broker to come up with the same id. The consumer
relies on the broker id to resume consumption properly.

Thanks,

Jun

On Fri, Dec 9, 2011 at 8:34 AM, Florian Leibert <[EMAIL PROTECTED]> wrote:

> Hi -
> I see there's a static broker-id in the properties files. Is this optional?
> When using Zookeeper, couldn't ZK just give each broker if registered as a
> SEQ node a generated ID that's guaranteed to be unique? Otherwise can this
> ID be a randomly generated ID? Basically I'd like to bring up instances in
> EC2 without having to worry about the state of other brokers...
>
> Thanks!
> Flo
>

But this means that I can pick one at random - it doesn't have to be
sequential as long as it's the same after restart?

On Fri, Dec 9, 2011 at 9:37 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Florian,
>
> The broker id uniquely identifies a broker. Every time a broker is
> restarted, we want the broker to come up with the same id. The consumer
> relies on the broker id to resume consumption properly.
>
> Thanks,
>
> Jun
>
> On Fri, Dec 9, 2011 at 8:34 AM, Florian Leibert <[EMAIL PROTECTED]> wrote:
>
> > Hi -
> > I see there's a static broker-id in the properties files. Is this
> optional?
> > When using Zookeeper, couldn't ZK just give each broker if registered as
> a
> > SEQ node a generated ID that's guaranteed to be unique? Otherwise can
> this
> > ID be a randomly generated ID? Basically I'd like to bring up instances
> in
> > EC2 without having to worry about the state of other brokers...
> >
> > Thanks!
> > Flo
> >
>

--
Best regards,

Florian
http://twitter.com/flo <http://twitter.com/floleibert>
http://flori.posterous.com/

I think what Florian is saying is actually subtly different. Usually people
ask, "Do you really need a node id? why not just use host/port so I don't
need to allocate an id?" As Jun says you need a node id so that data is
portable and not tied to the host/port of the machine serving it. But if we
rephrase what Florian is saying as a feature request I would put it this
way, it is kind of a pain to have to allocate ids "by hand" and this makes
it harder to add/remove brokers easily. The right id to assign, after all,
is always the next unused integer. It could instead be that node ids are
auto-allocated when the broker first starts and stored with the data. The
mechanism would be a file in the data directory called .node-id. If this
file is absent the broker would allocate itself the next available id in zk
in a transactional way and store it in the .node-id file and assume that
node id from then on. This would make dynamic deployment on ec2 or for
integration/perf tests much easier, and would likely ease our own
configuration as well since we would not have any per-machine config
overrides.

The question is, how would this interact with replication...?

-Jay

On Fri, Dec 9, 2011 at 9:37 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Florian,
>
> The broker id uniquely identifies a broker. Every time a broker is
> restarted, we want the broker to come up with the same id. The consumer
> relies on the broker id to resume consumption properly.
>
> Thanks,
>
> Jun
>
> On Fri, Dec 9, 2011 at 8:34 AM, Florian Leibert <[EMAIL PROTECTED]> wrote:
>
> > Hi -
> > I see there's a static broker-id in the properties files. Is this
> optional?
> > When using Zookeeper, couldn't ZK just give each broker if registered as
> a
> > SEQ node a generated ID that's guaranteed to be unique? Otherwise can
> this
> > ID be a randomly generated ID? Basically I'd like to bring up instances
> in
> > EC2 without having to worry about the state of other brokers...
> >
> > Thanks!
> > Flo
> >
>

Yes, I don't think they need to be sequential though we always assign them
sequentially at LinkedIn.

-Jay

On Fri, Dec 9, 2011 at 9:46 AM, Florian Leibert <[EMAIL PROTECTED]> wrote:

> But this means that I can pick one at random - it doesn't have to be
> sequential as long as it's the same after restart?
>
> On Fri, Dec 9, 2011 at 9:37 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Florian,
> >
> > The broker id uniquely identifies a broker. Every time a broker is
> > restarted, we want the broker to come up with the same id. The consumer
> > relies on the broker id to resume consumption properly.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Dec 9, 2011 at 8:34 AM, Florian Leibert <[EMAIL PROTECTED]> wrote:
> >
> > > Hi -
> > > I see there's a static broker-id in the properties files. Is this
> > optional?
> > > When using Zookeeper, couldn't ZK just give each broker if registered
> as
> > a
> > > SEQ node a generated ID that's guaranteed to be unique? Otherwise can
> > this
> > > ID be a randomly generated ID? Basically I'd like to bring up instances
> > in
> > > EC2 without having to worry about the state of other brokers...
> > >
> > > Thanks!
> > > Flo
> > >
> >
>
>
>
> --
> Best regards,
>
> Florian
> http://twitter.com/flo <http://twitter.com/floleibert>
> http://flori.posterous.com/
>

Florian,

>> When using Zookeeper, couldn't ZK just give each broker if registered as
a
SEQ node a generated ID that's guaranteed to be unique? Otherwise can this
ID be a randomly generated ID?

If you do that, then what happens when you merely bounce the broker ? It
might end up getting a different id each time, which will break consumer
logic.

The purpose of a unique broker id is to be able to identify a Kafka broker,
without having to know its host-name/IP. This allows you to switch the
machine hosting a broker, without changing any client config.

Thanks,
Neha

On Fri, Dec 9, 2011 at 9:37 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Florian,
>
> The broker id uniquely identifies a broker. Every time a broker is
> restarted, we want the broker to come up with the same id. The consumer
> relies on the broker id to resume consumption properly.
>
> Thanks,
>
> Jun
>
> On Fri, Dec 9, 2011 at 8:34 AM, Florian Leibert <[EMAIL PROTECTED]> wrote:
>
> > Hi -
> > I see there's a static broker-id in the properties files. Is this
> optional?
> > When using Zookeeper, couldn't ZK just give each broker if registered as
> a
> > SEQ node a generated ID that's guaranteed to be unique? Otherwise can
> this
> > ID be a randomly generated ID? Basically I'd like to bring up instances
> in
> > EC2 without having to worry about the state of other brokers...
> >
> > Thanks!
> > Flo
> >
>

Right but if this is just part of the node allocation script that would
work. E.g. if your allocate_node.sh script was something like:

cp default_server_config.properties server.properties
echo "node.id=" random_number() >> server.properties
host=allocate_ec2_node()
scp -r kafka host:
scp -r server.properties host:
ssh host kafka/bin/kafka-server-start.sh server.properties

Bouncing that node would retain the node id from then on but it initially
starts with a random id which avoids needing to maintain a counter to know
which ids are currently in use.

-Jay

On Fri, Dec 9, 2011 at 9:55 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Florian,
>
> >> When using Zookeeper, couldn't ZK just give each broker if registered as
> a
> SEQ node a generated ID that's guaranteed to be unique? Otherwise can this
> ID be a randomly generated ID?
>
> If you do that, then what happens when you merely bounce the broker ? It
> might end up getting a different id each time, which will break consumer
> logic.
>
> The purpose of a unique broker id is to be able to identify a Kafka broker,
> without having to know its host-name/IP. This allows you to switch the
> machine hosting a broker, without changing any client config.
>
> Thanks,
> Neha
>
> On Fri, Dec 9, 2011 at 9:37 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Florian,
> >
> > The broker id uniquely identifies a broker. Every time a broker is
> > restarted, we want the broker to come up with the same id. The consumer
> > relies on the broker id to resume consumption properly.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Dec 9, 2011 at 8:34 AM, Florian Leibert <[EMAIL PROTECTED]> wrote:
> >
> > > Hi -
> > > I see there's a static broker-id in the properties files. Is this
> > optional?
> > > When using Zookeeper, couldn't ZK just give each broker if registered
> as
> > a
> > > SEQ node a generated ID that's guaranteed to be unique? Otherwise can
> > this
> > > ID be a randomly generated ID? Basically I'd like to bring up instances
> > in
> > > EC2 without having to worry about the state of other brokers...
> > >
> > > Thanks!
> > > Flo
> > >
> >
>

Yes, I believe that would work.

Thanks,
Neha

On Fri, Dec 9, 2011 at 10:05 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Right but if this is just part of the node allocation script that would
> work. E.g. if your allocate_node.sh script was something like:
>
> cp default_server_config.properties server.properties
> echo "node.id=" random_number() >> server.properties
> host=allocate_ec2_node()
> scp -r kafka host:
> scp -r server.properties host:
> ssh host kafka/bin/kafka-server-start.sh server.properties
>
> Bouncing that node would retain the node id from then on but it initially
> starts with a random id which avoids needing to maintain a counter to know
> which ids are currently in use.
>
> -Jay
>
> On Fri, Dec 9, 2011 at 9:55 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > Florian,
> >
> > >> When using Zookeeper, couldn't ZK just give each broker if registered
> as
> > a
> > SEQ node a generated ID that's guaranteed to be unique? Otherwise can
> this
> > ID be a randomly generated ID?
> >
> > If you do that, then what happens when you merely bounce the broker ? It
> > might end up getting a different id each time, which will break consumer
> > logic.
> >
> > The purpose of a unique broker id is to be able to identify a Kafka
> broker,
> > without having to know its host-name/IP. This allows you to switch the
> > machine hosting a broker, without changing any client config.
> >
> > Thanks,
> > Neha
> >
> > On Fri, Dec 9, 2011 at 9:37 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Florian,
> > >
> > > The broker id uniquely identifies a broker. Every time a broker is
> > > restarted, we want the broker to come up with the same id. The consumer
> > > relies on the broker id to resume consumption properly.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Dec 9, 2011 at 8:34 AM, Florian Leibert <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > Hi -
> > > > I see there's a static broker-id in the properties files. Is this
> > > optional?
> > > > When using Zookeeper, couldn't ZK just give each broker if registered
> > as
> > > a
> > > > SEQ node a generated ID that's guaranteed to be unique? Otherwise can
> > > this
> > > > ID be a randomly generated ID? Basically I'd like to bring up
> instances
> > > in
> > > > EC2 without having to worry about the state of other brokers...
> > > >
> > > > Thanks!
> > > > Flo
> > > >
> > >
> >
>

On 12/09/2011 12:48 PM, Jay Kreps wrote:
> Yes, I don't think they need to be sequential though we always assign them
> sequentially at LinkedIn.

At Clearspring we found it convenient to uses the last octet of the IP
since that was already unique per cluster (no clusters span multiple
subnets [1]).  We have on occasion confused ourselves by having
non-sequential ids, but it's probably best that those bugs were squashed
sooner rather than latter.

[1] And yes, this will be a nifty problem when a cluster no longer fits
on a single subnet ;-)