Hi,

I am searching for a Kafka Producer that is able to write to a Kafka Cluster with replication, so that there is no data lost when one broker is going down. This functionality has been added in Kafka 0.8 as far as I know. In the current (0.6) configuration we setup the Producer as a Perl Module inside a nginx web server to handle HTTP requests. I am trying to achieve a similar functionality, but with a Kafka-Cluster/ZooKeeper combination instead of a single Kafka server behind it.

What would be the best way to achieve this? I haven't found a Perl Module that can talk to a Kafka/ZooKeeper configuration so far.

Regards,
Tobias
 

In 0.8, the producer protocol has changed a bit and we don't have a perl
implementation yet. The following wiki is intended for people to build
producers in languages other than java.

https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol

Thanks,

Jun

On Wed, Dec 19, 2012 at 2:02 AM, Tobias Feldhaus
<[EMAIL PROTECTED]>wrote:

> Hi,
>
> I am searching for a Kafka Producer that is able to write to a Kafka
> Cluster with replication, so that there is no data lost when one broker is
> going down. This functionality has been added in Kafka 0.8 as far as I
> know. In the current (0.6) configuration we setup the Producer as a Perl
> Module inside a nginx web server to handle HTTP requests. I am trying to
> achieve a similar functionality, but with a Kafka-Cluster/ZooKeeper
> combination instead of a single Kafka server behind it.
>
> What would be the best way to achieve this? I haven't found a Perl Module
> that can talk to a Kafka/ZooKeeper configuration so far.
>
> Regards,
> Tobias

 

Hi  Tobias

I am not using a nginx but a tomcat server, but I solved the same
problem in this tomcat server.

I created a Logback appender that writes to Kafka,

My tomcat server uses logback to log each of the requests,

I configured this logback to write to a file, and to write to a kafka
cluster (using the mentioned appender)

If you like, I can try to share my code,
On 12/19/2012 12:02 PM, Tobias Feldhaus wrote:
> Hi,
>
> I am searching for a Kafka Producer that is able to write to a Kafka Cluster with replication, so that there is no data lost when one broker is going down. This functionality has been added in Kafka 0.8 as far as I know. In the current (0.6) configuration we setup the Producer as a Perl Module inside a nginx web server to handle HTTP requests. I am trying to achieve a similar functionality, but with a Kafka-Cluster/ZooKeeper combination instead of a single Kafka server behind it.
>
> What would be the best way to achieve this? I haven't found a Perl Module that can talk to a Kafka/ZooKeeper configuration so far.
>
> Regards,
> Tobias