Can you do the following -

1. Start a mirror Kafka cluster with the new version on a separate
zookeeper namespace. Configure this to mirror data from the existing kafka
cluster.
2. Move your consumers to pull data from the mirror
3. For each producer, one at a time, change the zookeeper namespace to
point to the mirror and restart the producer.
4. Once the producers have moved to mirror cluster, shutdown mirroring and
old cluster.

Thanks,
Neha

On Tuesday, March 19, 2013, Jason Rosenberg wrote:

> I need to upgrade some kafka broker servers.  So I need to seamlessly
> migrate traffic from the old brokers to the new ones, without losing data,
> and without stopping producers.  I can temporarily stop consumers, etc.
>
> Is there a strategy for this?
>
> Also, because of the way we are embedding kafka in our framework, our
> brokerId's are auto-generated (based on hostname, etc.), so I can't simply
> copy over broker log files, etc., by transferring an old brokerId to a new
> host.
>
> Is there a way to change the view of the cluster from the producer's
> standpoint, without doing so from the consumers standpoint?  That way, the
> producers can start writing to the new brokers, while the consumers drain
> all data from the old brokers before switching to the new brokers.
>
> I don't actually care about ordering of messages, since the consumers are
> publishing them to a store that will index them properly based on source
> timestamp, etc.
>
> We are using zk for both producers and consumers connections.
>
> This is using 0.7.2.  I assume in 0.8 it will be easier, since with
> replication, you can phase in the new servers gradually, etc., no?
>
> Thanks,
>
> Jason
>

 

I can do most of that I presume.

It looks like to set up a separate namespace for zk, I can add /path at the
end of each node:port in my zkconnect string, e.g.:
 zkhost1:123/newnamespace,zkhost2:123/newnamespace
right?

For mirroring, there's some vague documentation here:
https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
Is this the most up to date approach for 0.7.2?  Set up a MirrorMaker
intermediate process that consumes from the old and produces to the new?

I am not able to restart producers one by one (as there are many, on a
rather asynchronous update/restart cycle).  But I can eventually get them
migrated over, etc.

Jason

On Tue, Mar 19, 2013 at 7:07 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Can you do the following -
>
> 1. Start a mirror Kafka cluster with the new version on a separate
> zookeeper namespace. Configure this to mirror data from the existing kafka
> cluster.
> 2. Move your consumers to pull data from the mirror
> 3. For each producer, one at a time, change the zookeeper namespace to
> point to the mirror and restart the producer.
> 4. Once the producers have moved to mirror cluster, shutdown mirroring and
> old cluster.
>
> Thanks,
> Neha
>
> On Tuesday, March 19, 2013, Jason Rosenberg wrote:
>
> > I need to upgrade some kafka broker servers.  So I need to seamlessly
> > migrate traffic from the old brokers to the new ones, without losing
> data,
> > and without stopping producers.  I can temporarily stop consumers, etc.
> >
> > Is there a strategy for this?
> >
> > Also, because of the way we are embedding kafka in our framework, our
> > brokerId's are auto-generated (based on hostname, etc.), so I can't
> simply
> > copy over broker log files, etc., by transferring an old brokerId to a
> new
> > host.
> >
> > Is there a way to change the view of the cluster from the producer's
> > standpoint, without doing so from the consumers standpoint?  That way,
> the
> > producers can start writing to the new brokers, while the consumers drain
> > all data from the old brokers before switching to the new brokers.
> >
> > I don't actually care about ordering of messages, since the consumers are
> > publishing them to a store that will index them properly based on source
> > timestamp, etc.
> >
> > We are using zk for both producers and consumers connections.
> >
> > This is using 0.7.2.  I assume in 0.8 it will be easier, since with
> > replication, you can phase in the new servers gradually, etc., no?
> >
> > Thanks,
> >
> > Jason
> >
>

 

The zookeeper connection URL with namespace can be
zkhost1:123,zkhost2:123/newnamespace

The wiki is up to date for Kafka 0.7.2. There is no officially supported
feature to do that sort of migration, I suggested one approach that I could
think of :-)

Thanks,
Neha

On Tuesday, March 19, 2013, Jason Rosenberg wrote:

> I can do most of that I presume.
>
> It looks like to set up a separate namespace for zk, I can add /path at the
> end of each node:port in my zkconnect string, e.g.:
>  zkhost1:123/newnamespace,zkhost2:123/newnamespace
> right?
>
> For mirroring, there's some vague documentation here:
> https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
> Is this the most up to date approach for 0.7.2?  Set up a MirrorMaker
> intermediate process that consumes from the old and produces to the new?
>
> I am not able to restart producers one by one (as there are many, on a
> rather asynchronous update/restart cycle).  But I can eventually get them
> migrated over, etc.
>
> Jason
>
> On Tue, Mar 19, 2013 at 7:07 PM, Neha Narkhede <[EMAIL PROTECTED]<javascript:;>
> >wrote:
>
> > Can you do the following -
> >
> > 1. Start a mirror Kafka cluster with the new version on a separate
> > zookeeper namespace. Configure this to mirror data from the existing
> kafka
> > cluster.
> > 2. Move your consumers to pull data from the mirror
> > 3. For each producer, one at a time, change the zookeeper namespace to
> > point to the mirror and restart the producer.
> > 4. Once the producers have moved to mirror cluster, shutdown mirroring
> and
> > old cluster.
> >
> > Thanks,
> > Neha
> >
> > On Tuesday, March 19, 2013, Jason Rosenberg wrote:
> >
> > > I need to upgrade some kafka broker servers.  So I need to seamlessly
> > > migrate traffic from the old brokers to the new ones, without losing
> > data,
> > > and without stopping producers.  I can temporarily stop consumers, etc.
> > >
> > > Is there a strategy for this?
> > >
> > > Also, because of the way we are embedding kafka in our framework, our
> > > brokerId's are auto-generated (based on hostname, etc.), so I can't
> > simply
> > > copy over broker log files, etc., by transferring an old brokerId to a
> > new
> > > host.
> > >
> > > Is there a way to change the view of the cluster from the producer's
> > > standpoint, without doing so from the consumers standpoint?  That way,
> > the
> > > producers can start writing to the new brokers, while the consumers
> drain
> > > all data from the old brokers before switching to the new brokers.
> > >
> > > I don't actually care about ordering of messages, since the consumers
> are
> > > publishing them to a store that will index them properly based on
> source
> > > timestamp, etc.
> > >
> > > We are using zk for both producers and consumers connections.
> > >
> > > This is using 0.7.2.  I assume in 0.8 it will be easier, since with
> > > replication, you can phase in the new servers gradually, etc., no?
> > >
> > > Thanks,
> > >
> > > Jason
> > >
> >
>

 

I think might be cool, would be to have a feature where by you can tell a
broker to stop accepting new data produced to it, but still allow consumers
to consume from it.

That way, you can roll out new brokers to a cluster, turn off producing to
the old nodes, then wait for the log retention period, and then remove the
old nodes from the cluster.

Does that make sense?  Could it be easily done?

Or is all this a non-issue anyway in 0.8?

Maybe I should just wait for 0.8 to be ready, before doing my migration
anyway.....

Jason

On Wed, Mar 20, 2013 at 6:42 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> The zookeeper connection URL with namespace can be
> zkhost1:123,zkhost2:123/newnamespace
>
> The wiki is up to date for Kafka 0.7.2. There is no officially supported
> feature to do that sort of migration, I suggested one approach that I could
> think of :-)
>
> Thanks,
> Neha
>
> On Tuesday, March 19, 2013, Jason Rosenberg wrote:
>
> > I can do most of that I presume.
> >
> > It looks like to set up a separate namespace for zk, I can add /path at
> the
> > end of each node:port in my zkconnect string, e.g.:
> >  zkhost1:123/newnamespace,zkhost2:123/newnamespace
> > right?
> >
> > For mirroring, there's some vague documentation here:
> > https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
> > Is this the most up to date approach for 0.7.2?  Set up a MirrorMaker
> > intermediate process that consumes from the old and produces to the new?
> >
> > I am not able to restart producers one by one (as there are many, on a
> > rather asynchronous update/restart cycle).  But I can eventually get them
> > migrated over, etc.
> >
> > Jason
> >
> > On Tue, Mar 19, 2013 at 7:07 PM, Neha Narkhede <[EMAIL PROTECTED]
> <javascript:;>
> > >wrote:
> >
> > > Can you do the following -
> > >
> > > 1. Start a mirror Kafka cluster with the new version on a separate
> > > zookeeper namespace. Configure this to mirror data from the existing
> > kafka
> > > cluster.
> > > 2. Move your consumers to pull data from the mirror
> > > 3. For each producer, one at a time, change the zookeeper namespace to
> > > point to the mirror and restart the producer.
> > > 4. Once the producers have moved to mirror cluster, shutdown mirroring
> > and
> > > old cluster.
> > >
> > > Thanks,
> > > Neha
> > >
> > > On Tuesday, March 19, 2013, Jason Rosenberg wrote:
> > >
> > > > I need to upgrade some kafka broker servers.  So I need to seamlessly
> > > > migrate traffic from the old brokers to the new ones, without losing
> > > data,
> > > > and without stopping producers.  I can temporarily stop consumers,
> etc.
> > > >
> > > > Is there a strategy for this?
> > > >
> > > > Also, because of the way we are embedding kafka in our framework, our
> > > > brokerId's are auto-generated (based on hostname, etc.), so I can't
> > > simply
> > > > copy over broker log files, etc., by transferring an old brokerId to
> a
> > > new
> > > > host.
> > > >
> > > > Is there a way to change the view of the cluster from the producer's
> > > > standpoint, without doing so from the consumers standpoint?  That
> way,
> > > the
> > > > producers can start writing to the new brokers, while the consumers
> > drain
> > > > all data from the old brokers before switching to the new brokers.
> > > >
> > > > I don't actually care about ordering of messages, since the consumers
> > are
> > > > publishing them to a store that will index them properly based on
> > source
> > > > timestamp, etc.
> > > >
> > > > We are using zk for both producers and consumers connections.
> > > >
> > > > This is using 0.7.2.  I assume in 0.8 it will be easier, since with
> > > > replication, you can phase in the new servers gradually, etc., no?
> > > >
> > > > Thanks,
> > > >
> > > > Jason
> > > >
> > >
> >
>

 

Makes sense. This is a non issue in 0.8, see this -
https://issues.apache.org/jira/browse/KAFKA-155#comment-13607766

Thanks,
Neha
On Wed, Mar 20, 2013 at 8:41 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> I think might be cool, would be to have a feature where by you can tell a
> broker to stop accepting new data produced to it, but still allow consumers
> to consume from it.
>
> That way, you can roll out new brokers to a cluster, turn off producing to
> the old nodes, then wait for the log retention period, and then remove the
> old nodes from the cluster.
>
> Does that make sense?  Could it be easily done?
>
> Or is all this a non-issue anyway in 0.8?
>
> Maybe I should just wait for 0.8 to be ready, before doing my migration
> anyway.....
>
> Jason
>
> On Wed, Mar 20, 2013 at 6:42 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > The zookeeper connection URL with namespace can be
> > zkhost1:123,zkhost2:123/newnamespace
> >
> > The wiki is up to date for Kafka 0.7.2. There is no officially supported
> > feature to do that sort of migration, I suggested one approach that I
> could
> > think of :-)
> >
> > Thanks,
> > Neha
> >
> > On Tuesday, March 19, 2013, Jason Rosenberg wrote:
> >
> > > I can do most of that I presume.
> > >
> > > It looks like to set up a separate namespace for zk, I can add /path at
> > the
> > > end of each node:port in my zkconnect string, e.g.:
> > >  zkhost1:123/newnamespace,zkhost2:123/newnamespace
> > > right?
> > >
> > > For mirroring, there's some vague documentation here:
> > > https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
> > > Is this the most up to date approach for 0.7.2?  Set up a MirrorMaker
> > > intermediate process that consumes from the old and produces to the
> new?
> > >
> > > I am not able to restart producers one by one (as there are many, on a
> > > rather asynchronous update/restart cycle).  But I can eventually get
> them
> > > migrated over, etc.
> > >
> > > Jason
> > >
> > > On Tue, Mar 19, 2013 at 7:07 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > <javascript:;>
> > > >wrote:
> > >
> > > > Can you do the following -
> > > >
> > > > 1. Start a mirror Kafka cluster with the new version on a separate
> > > > zookeeper namespace. Configure this to mirror data from the existing
> > > kafka
> > > > cluster.
> > > > 2. Move your consumers to pull data from the mirror
> > > > 3. For each producer, one at a time, change the zookeeper namespace
> to
> > > > point to the mirror and restart the producer.
> > > > 4. Once the producers have moved to mirror cluster, shutdown
> mirroring
> > > and
> > > > old cluster.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > > On Tuesday, March 19, 2013, Jason Rosenberg wrote:
> > > >
> > > > > I need to upgrade some kafka broker servers.  So I need to
> seamlessly
> > > > > migrate traffic from the old brokers to the new ones, without
> losing
> > > > data,
> > > > > and without stopping producers.  I can temporarily stop consumers,
> > etc.
> > > > >
> > > > > Is there a strategy for this?
> > > > >
> > > > > Also, because of the way we are embedding kafka in our framework,
> our
> > > > > brokerId's are auto-generated (based on hostname, etc.), so I can't
> > > > simply
> > > > > copy over broker log files, etc., by transferring an old brokerId
> to
> > a
> > > > new
> > > > > host.
> > > > >
> > > > > Is there a way to change the view of the cluster from the
> producer's
> > > > > standpoint, without doing so from the consumers standpoint?  That
> > way,
> > > > the
> > > > > producers can start writing to the new brokers, while the consumers
> > > drain
> > > > > all data from the old brokers before switching to the new brokers.
> > > > >
> > > > > I don't actually care about ordering of messages, since the
> consumers
> > > are
> > > > > publishing them to a store that will index them properly based on
> > > source
> > > > > timestamp, etc.
> > > > >
> > > > > We are using zk for both producers and consumers connections.

 

On Wed, Mar 20, 2013 at 8:41 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> I think might be cool, would be to have a feature where by you can tell a
> broker to stop accepting new data produced to it, but still allow consumers
> to consume from it.
>
> That way, you can roll out new brokers to a cluster, turn off producing to
> the old nodes, then wait for the log retention period, and then remove the
> old nodes from the cluster.
>
> Does that make sense?  Could it be easily done?

Does to me. A way to do this is to place a load-balancer between your
consumers and brokers, allowing individual brokers to be taken out of
rotation for maintenance.

>
> Or is all this a non-issue anyway in 0.8?
>
> Maybe I should just wait for 0.8 to be ready, before doing my migration
> anyway.....
>
> Jason
>
> On Wed, Mar 20, 2013 at 6:42 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> The zookeeper connection URL with namespace can be
>> zkhost1:123,zkhost2:123/newnamespace
>>
>> The wiki is up to date for Kafka 0.7.2. There is no officially supported
>> feature to do that sort of migration, I suggested one approach that I could
>> think of :-)
>>
>> Thanks,
>> Neha
>>
>> On Tuesday, March 19, 2013, Jason Rosenberg wrote:
>>
>> > I can do most of that I presume.
>> >
>> > It looks like to set up a separate namespace for zk, I can add /path at
>> the
>> > end of each node:port in my zkconnect string, e.g.:
>> >  zkhost1:123/newnamespace,zkhost2:123/newnamespace
>> > right?
>> >
>> > For mirroring, there's some vague documentation here:
>> > https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
>> > Is this the most up to date approach for 0.7.2?  Set up a MirrorMaker
>> > intermediate process that consumes from the old and produces to the new?
>> >
>> > I am not able to restart producers one by one (as there are many, on a
>> > rather asynchronous update/restart cycle).  But I can eventually get them
>> > migrated over, etc.
>> >
>> > Jason
>> >
>> > On Tue, Mar 19, 2013 at 7:07 PM, Neha Narkhede <[EMAIL PROTECTED]
>> <javascript:;>
>> > >wrote:
>> >
>> > > Can you do the following -
>> > >
>> > > 1. Start a mirror Kafka cluster with the new version on a separate
>> > > zookeeper namespace. Configure this to mirror data from the existing
>> > kafka
>> > > cluster.
>> > > 2. Move your consumers to pull data from the mirror
>> > > 3. For each producer, one at a time, change the zookeeper namespace to
>> > > point to the mirror and restart the producer.
>> > > 4. Once the producers have moved to mirror cluster, shutdown mirroring
>> > and
>> > > old cluster.
>> > >
>> > > Thanks,
>> > > Neha
>> > >
>> > > On Tuesday, March 19, 2013, Jason Rosenberg wrote:
>> > >
>> > > > I need to upgrade some kafka broker servers.  So I need to seamlessly
>> > > > migrate traffic from the old brokers to the new ones, without losing
>> > > data,
>> > > > and without stopping producers.  I can temporarily stop consumers,
>> etc.
>> > > >
>> > > > Is there a strategy for this?
>> > > >
>> > > > Also, because of the way we are embedding kafka in our framework, our
>> > > > brokerId's are auto-generated (based on hostname, etc.), so I can't
>> > > simply
>> > > > copy over broker log files, etc., by transferring an old brokerId to
>> a
>> > > new
>> > > > host.
>> > > >
>> > > > Is there a way to change the view of the cluster from the producer's
>> > > > standpoint, without doing so from the consumers standpoint?  That
>> way,
>> > > the
>> > > > producers can start writing to the new brokers, while the consumers
>> > drain
>> > > > all data from the old brokers before switching to the new brokers.
>> > > >
>> > > > I don't actually care about ordering of messages, since the consumers
>> > are
>> > > > publishing them to a store that will index them properly based on
>> > source
>> > > > timestamp, etc.
>> > > >
>> > > > We are using zk for both producers and consumers connections.
>> > > >
>> > > > This is using 0.7.2.  I assume in 0.8 it will be easier, since with

 

On Wed, Mar 20, 2013 at 9:06 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:

> On Wed, Mar 20, 2013 at 8:41 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> > I think might be cool, would be to have a feature where by you can tell a
> > broker to stop accepting new data produced to it, but still allow
> consumers
> > to consume from it.
> >
> > That way, you can roll out new brokers to a cluster, turn off producing
> to
> > the old nodes, then wait for the log retention period, and then remove
> the
> > old nodes from the cluster.
> >
> > Does that make sense?  Could it be easily done?
>
> Does to me. A way to do this is to place a load-balancer between your
> consumers and brokers, allowing individual brokers to be taken out of
> rotation for maintenance.
Not sure I understand.  I need for a node to be taken out of the pool that
producers produce to, but still need consumers consuming from all brokers,
while we drain data from the brokers to be replaced.

Since I am using zk for producers to discover brokers, there's not an easy
way to tell producers to stop producing to a sub-set of nodes, without also
having the same affect on consumers.

Maybe I should first switch all producers to use a brokerlist, pointing to
only the new hosts.  But still have all the hosts in zk.....that might
work, I should think.  But then I'm making the commitment not to be using
zk to connect to the brokers for producers.

Jason

 

On Wed, Mar 20, 2013 at 10:55 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> On Wed, Mar 20, 2013 at 9:06 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>
>> On Wed, Mar 20, 2013 at 8:41 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>> > I think might be cool, would be to have a feature where by you can tell a
>> > broker to stop accepting new data produced to it, but still allow
>> consumers
>> > to consume from it.
>> >
>> > That way, you can roll out new brokers to a cluster, turn off producing
>> to
>> > the old nodes, then wait for the log retention period, and then remove
>> the
>> > old nodes from the cluster.
>> >
>> > Does that make sense?  Could it be easily done?
>>
>> Does to me. A way to do this is to place a load-balancer between your
>> consumers and brokers, allowing individual brokers to be taken out of
>> rotation for maintenance.
>
>
> Not sure I understand.  I need for a node to be taken out of the pool that
> producers produce to, but still need consumers consuming from all brokers,
> while we drain data from the brokers to be replaced.

Taking it out from behind the load-balancer between Producers and
Kafka means that Producers can no longer write to it. I said nothing
about disconnecting the *Consumers*. :-)

Philip

 

On Wed, Mar 20, 2013 at 11:10 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:

> On Wed, Mar 20, 2013 at 10:55 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
>
> > On Wed, Mar 20, 2013 at 9:06 AM, Philip O'Toole <[EMAIL PROTECTED]>
> wrote:
> >
> >> On Wed, Mar 20, 2013 at 8:41 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >> > I think might be cool, would be to have a feature where by you can
> tell a
> >> > broker to stop accepting new data produced to it, but still allow
> >> consumers
> >> > to consume from it.
> >> >
> >> > That way, you can roll out new brokers to a cluster, turn off
> producing
> >> to
> >> > the old nodes, then wait for the log retention period, and then remove
> >> the
> >> > old nodes from the cluster.
> >> >
> >> > Does that make sense?  Could it be easily done?
> >>
> >> Does to me. A way to do this is to place a load-balancer between your
> >> consumers and brokers, allowing individual brokers to be taken out of
> >> rotation for maintenance.
> >
> >
> > Not sure I understand.  I need for a node to be taken out of the pool
> that
> > producers produce to, but still need consumers consuming from all
> brokers,
> > while we drain data from the brokers to be replaced.
>
> Taking it out from behind the load-balancer between Producers and
> Kafka means that Producers can no longer write to it. I said nothing
> about disconnecting the *Consumers*. :-)
>
>
No worries Philip, I'll assume you you mispoke at first when talking about
a load-balancer between the consumers and brokers.  Kafka, unfortunately,
doesn't allow consumers to connect to kafka via a load balancer.  For
producers, also, you can't really use a load-balancer to connect to brokers
(you can use zk, or you can use a broker list, in 0.7.2, and in 0.8, you
can use an LB for the initial meta data connection, but then you still have
to have direct connections to each broker from each producer).

Jason

 

> No worries Philip, I'll assume you you mispoke at first when talking about
> a load-balancer between the consumers and brokers.  Kafka, unfortunately,
> doesn't allow consumers to connect to kafka via a load balancer.

Ah yes, I misspoke. I meant an LB between Producers and Kafka Brokers.

> For
> producers, also, you can't really use a load-balancer to connect to brokers
> (you can use zk, or you can use a broker list, in 0.7.2, and in 0.8, you
> can use an LB for the initial meta data connection, but then you still have
> to have direct connections to each broker from each producer).

Huh? Sure you can, if the Producers are simple. Producers just need a
destination IP address and port. They have no way of knowing if that
IP is an LB or real Kafka broker. Set the partition to -1 (i.e.
random) in all messages destined for Kafka, and it all just works.
Granted, this is a simple type of 0.72 deployment. Perhaps it doesn't
work for more complex Producers or 0.8 deployments -- I am not yet
that familiar with 0.8.

Philip

 

On Wed, Mar 20, 2013 at 12:00 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:

> > For
> > producers, also, you can't really use a load-balancer to connect to
> brokers
> > (you can use zk, or you can use a broker list, in 0.7.2, and in 0.8, you
> > can use an LB for the initial meta data connection, but then you still
> have
> > to have direct connections to each broker from each producer).
>
> Huh? Sure you can, if the Producers are simple. Producers just need a
> destination IP address and port. They have no way of knowing if that
> IP is an LB or real Kafka broker. Set the partition to -1 (i.e.
> random) in all messages destined for Kafka, and it all just works.
> Granted, this is a simple type of 0.72 deployment. Perhaps it doesn't
> work for more complex Producers or 0.8 deployments -- I am not yet
> that familiar with 0.8.
Interesting, so in this case, do you just provide the vip in the
"broker.list" property for the producer?  That might just work for my needs.

Jason

 

On Wed, Mar 20, 2013 at 12:06 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> On Wed, Mar 20, 2013 at 12:00 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>
>> > For
>> > producers, also, you can't really use a load-balancer to connect to
>> brokers
>> > (you can use zk, or you can use a broker list, in 0.7.2, and in 0.8, you
>> > can use an LB for the initial meta data connection, but then you still
>> have
>> > to have direct connections to each broker from each producer).
>>
>> Huh? Sure you can, if the Producers are simple. Producers just need a
>> destination IP address and port. They have no way of knowing if that
>> IP is an LB or real Kafka broker. Set the partition to -1 (i.e.
>> random) in all messages destined for Kafka, and it all just works.
>> Granted, this is a simple type of 0.72 deployment. Perhaps it doesn't
>> work for more complex Producers or 0.8 deployments -- I am not yet
>> that familiar with 0.8.
>
>
> Interesting, so in this case, do you just provide the vip in the
> "broker.list" property for the producer?  That might just work for my needs.

We have custom non-Java Producers so don't have Property files, but in
principle yes, that is what I mean.

Philip

 

Ok,

I programmatically configure my kafka producers, but essentially configure
them passing a set of config properties, as if specified in a .properties
file.

I'll think about trying this, seems like it might just work.

Jason
On Wed, Mar 20, 2013 at 12:16 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:

> On Wed, Mar 20, 2013 at 12:06 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> > On Wed, Mar 20, 2013 at 12:00 PM, Philip O'Toole <[EMAIL PROTECTED]>
> wrote:
> >
> >> > For
> >> > producers, also, you can't really use a load-balancer to connect to
> >> brokers
> >> > (you can use zk, or you can use a broker list, in 0.7.2, and in 0.8,
> you
> >> > can use an LB for the initial meta data connection, but then you still
> >> have
> >> > to have direct connections to each broker from each producer).
> >>
> >> Huh? Sure you can, if the Producers are simple. Producers just need a
> >> destination IP address and port. They have no way of knowing if that
> >> IP is an LB or real Kafka broker. Set the partition to -1 (i.e.
> >> random) in all messages destined for Kafka, and it all just works.
> >> Granted, this is a simple type of 0.72 deployment. Perhaps it doesn't
> >> work for more complex Producers or 0.8 deployments -- I am not yet
> >> that familiar with 0.8.
> >
> >
> > Interesting, so in this case, do you just provide the vip in the
> > "broker.list" property for the producer?  That might just work for my
> needs.
>
> We have custom non-Java Producers so don't have Property files, but in
> principle yes, that is what I mean.
>
> Philip
>