Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> A (nginx_module/Perl) Producer that writes to a Kafka Cluster with Replication


+
Tobias Feldhaus 2012-12-19, 10:03
+
Jun Rao 2012-12-19, 15:47
Copy link to this message
-
Re: A (nginx_module/Perl) Producer that writes to a Kafka Cluster with Replication
Hi  Tobias

I am not using a nginx but a tomcat server, but I solved the same
problem in this tomcat server.

I created a Logback appender that writes to Kafka,

My tomcat server uses logback to log each of the requests,

I configured this logback to write to a file, and to write to a kafka
cluster (using the mentioned appender)

If you like, I can try to share my code,
On 12/19/2012 12:02 PM, Tobias Feldhaus wrote:
> Hi,
>
> I am searching for a Kafka Producer that is able to write to a Kafka Cluster with replication, so that there is no data lost when one broker is going down. This functionality has been added in Kafka 0.8 as far as I know. In the current (0.6) configuration we setup the Producer as a Perl Module inside a nginx web server to handle HTTP requests. I am trying to achieve a similar functionality, but with a Kafka-Cluster/ZooKeeper combination instead of a single Kafka server behind it.
>
> What would be the best way to achieve this? I haven't found a Perl Module that can talk to a Kafka/ZooKeeper configuration so far.
>
> Regards,
> Tobias
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB