Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> What are my options? (Ruby/Rails environment)


+
Mark 2013-08-29, 20:40
Copy link to this message
-
Re: What are my options? (Ruby/Rails environment)
I assume this for Kafka 0.7. One option is to use a VIP in front of the
brokers for load balancing.

Thanks,

Jun
On Thu, Aug 29, 2013 at 1:39 PM, Mark <[EMAIL PROTECTED]> wrote:

> We are thinking about using Kafka to collect events from our Rails
> application and I was hoping to get some input from the Kafka community.
>
> Currently the only gems available are:
>
> https://github.com/acrosa/kafka-rb
> https://github.com/bpot/poseidon  (Can't use since we are only running
> 1.8.7)
>
> Now neither of these integrate with Zookeeper so we are missing quite a
> few features:
>
> - Auto discovery of brokers
> - Auto discover of partitions for consumers
> - … fill in the  rest here, new to Kafka so don't know everything that is
> missing
>
> I was wondering what my best options are going forward with Kakfa? I think
> we have the following choices
>
> A) Instead of writing directly to Kafka from our application we can write
> our events/messages to some other source (Syslog, File, ?) and then have a
> separate Java process that reads these sources and writes to Kafka. This is
> a little annoying since we now have to worry about every machine also
> running the above separate process to write to Kafka.
>
> B) Work around the above limitations. Auto-discover of brokers is terrible
> since we don't foresee us adding/removing brokers that frequently. The lack
> of auto-discover of partitions is definitely a loss since we now have to
> know which broker/partition to read from at all times. Of course we can
> just write to Kafka using the above Gems and have our consumers written in
> another language.
>
> C)?
>
> Any thoughts/opinions?
>
> Thanks
>
> - M
>
>

 
+
Travis Brady 2013-08-30, 16:01
+
Jun Rao 2013-08-31, 03:45
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB