Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Broker bind address versus published hostname in ZooKeeper


Copy link to this message
-
Re: Broker bind address versus published hostname in ZooKeeper
Timothy Chen 2013-10-17, 22:20
Hi Roger,

That's exactly what I need in my end, and actually internally created a new
property called zkHost.name to publish a different host to zk. This is also
needed for deploying Kafka into Azure.

I also created zkHost.port since the internal and external ports that's
exposed might be different as well.

Tim
On Thu, Oct 17, 2013 at 3:13 PM, Roger Hoover <[EMAIL PROTECTED]>wrote:

> Hi all,
>
> I'm getting started experimenting with Kafka and ran into a configuration
> issue.
>
> Currently, in server.properties, you can configure host.name which gets
> used for two purposes: 1) to bind the socket 2) to publish the broker
> details to ZK for clients to use.
>
> There are times when these two settings need to be different. Here's an
> example. I want to setup Kafka brokers on OpenStack virtual machines in a
> private cloud but I need producers to connect from elsewhere on the
> internal corporate network. With OpenStack, the virtual machines are only
> exposed to DHCP addresses (typically RFC 1918 private addresses). You can
> assign "floating ips" to a virtual machine but it's forwarded using Network
> Address Translation and not exposed directly to the VM. Also, there's
> typically no DNS to provide hostname lookup. Hosts have names like
> "fubar.novalocal" that are not externally routable.
>
> Here's what I want. I want the broker to bind to the VM's private network
> IP but I want it to publish it's floating IP to ZooKeeper so that producers
> can publish to it.
>
> I propose a new optional parameter, "listen", which would allow you to
> specify the socket address to listen on. If not set, the parameter would
> default to host.name, which is the current behavior.
>
> #Publish the externally routable IP in ZK
> host.name = <floating ip>
> #Accept connections from any interface the VM knows about
> listen = *
>
> I'm assuming others will eventually have the same requirement so I've added
> a JIRA ticket.
>
> https://issues.apache.org/jira/browse/KAFKA-1092
>
> Thanks for your consideration.
>
> Cheers,
>
> Roger
>