Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> What are my options? (Ruby/Rails environment)


+
Mark 2013-08-29, 20:40
+
Jun Rao 2013-08-30, 03:45
+
Travis Brady 2013-08-30, 16:01
Copy link to this message
-
Re: What are my options? (Ruby/Rails environment)
In 0.8, the producer no longer depends on ZK. It only takes a list of
brokers. At LinkedIn, we have a 0.8 C producer implementation and plan to
open source it soon.

Thanks,

Jun
On Fri, Aug 30, 2013 at 9:00 AM, Travis Brady <[EMAIL PROTECTED]>wrote:

> I think this points out the need for a single canonical cross-platform C
> client lib with support for Zookeeper that could easily wrapped for use in
> other languages.
>
> It would make it much easier for people using Python, Ruby, Node, Lua,
> Haskell, Go, OCaml, etc to have such a library that matches the features of
> the Scala/Java client.
> Currently writing a client amounts to re-implementing the entire Scala
> client from scratch, which is why so many clients settle for simple
> producer/consumer support and skip Zk altogether.
>
>
>
>
> On Thu, Aug 29, 2013 at 10:44 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > I assume this for Kafka 0.7. One option is to use a VIP in front of the
> > brokers for load balancing.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Aug 29, 2013 at 1:39 PM, Mark <[EMAIL PROTECTED]> wrote:
> >
> > > We are thinking about using Kafka to collect events from our Rails
> > > application and I was hoping to get some input from the Kafka
> community.
> > >
> > > Currently the only gems available are:
> > >
> > > https://github.com/acrosa/kafka-rb
> > > https://github.com/bpot/poseidon  (Can't use since we are only running
> > > 1.8.7)
> > >
> > > Now neither of these integrate with Zookeeper so we are missing quite a
> > > few features:
> > >
> > > - Auto discovery of brokers
> > > - Auto discover of partitions for consumers
> > > - … fill in the  rest here, new to Kafka so don't know everything that
> is
> > > missing
> > >
> > > I was wondering what my best options are going forward with Kakfa? I
> > think
> > > we have the following choices
> > >
> > > A) Instead of writing directly to Kafka from our application we can
> write
> > > our events/messages to some other source (Syslog, File, ?) and then
> have
> > a
> > > separate Java process that reads these sources and writes to Kafka.
> This
> > is
> > > a little annoying since we now have to worry about every machine also
> > > running the above separate process to write to Kafka.
> > >
> > > B) Work around the above limitations. Auto-discover of brokers is
> > terrible
> > > since we don't foresee us adding/removing brokers that frequently. The
> > lack
> > > of auto-discover of partitions is definitely a loss since we now have
> to
> > > know which broker/partition to read from at all times. Of course we can
> > > just write to Kafka using the above Gems and have our consumers written
> > in
> > > another language.
> > >
> > > C)?
> > >
> > > Any thoughts/opinions?
> > >
> > > Thanks
> > >
> > > - M
> > >
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB