Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> What are my options? (Ruby/Rails environment)


+
Mark 2013-08-29, 20:40
+
Jun Rao 2013-08-30, 03:45
+
Travis Brady 2013-08-30, 16:01
Copy link to this message
-
Re: What are my options? (Ruby/Rails environment)
In 0.8, the producer no longer depends on ZK. It only takes a list of
brokers. At LinkedIn, we have a 0.8 C producer implementation and plan to
open source it soon.

Thanks,

Jun
On Fri, Aug 30, 2013 at 9:00 AM, Travis Brady <[EMAIL PROTECTED]>wrote:

> I think this points out the need for a single canonical cross-platform C
> client lib with support for Zookeeper that could easily wrapped for use in
> other languages.
>
> It would make it much easier for people using Python, Ruby, Node, Lua,
> Haskell, Go, OCaml, etc to have such a library that matches the features of
> the Scala/Java client.
> Currently writing a client amounts to re-implementing the entire Scala
> client from scratch, which is why so many clients settle for simple
> producer/consumer support and skip Zk altogether.
>
>
>
>
> On Thu, Aug 29, 2013 at 10:44 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > I assume this for Kafka 0.7. One option is to use a VIP in front of the
> > brokers for load balancing.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Aug 29, 2013 at 1:39 PM, Mark <[EMAIL PROTECTED]> wrote:
> >
> > > We are thinking about using Kafka to collect events from our Rails
> > > application and I was hoping to get some input from the Kafka
> community.
> > >
> > > Currently the only gems available are:
> > >
> > > https://github.com/acrosa/kafka-rb
> > > https://github.com/bpot/poseidon  (Can't use since we are only running
> > > 1.8.7)
> > >
> > > Now neither of these integrate with Zookeeper so we are missing quite a
> > > few features:
> > >
> > > - Auto discovery of brokers
> > > - Auto discover of partitions for consumers
> > > - … fill in the  rest here, new to Kafka so don't know everything that
> is
> > > missing
> > >
> > > I was wondering what my best options are going forward with Kakfa? I
> > think
> > > we have the following choices
> > >
> > > A) Instead of writing directly to Kafka from our application we can
> write
> > > our events/messages to some other source (Syslog, File, ?) and then
> have
> > a
> > > separate Java process that reads these sources and writes to Kafka.
> This
> > is
> > > a little annoying since we now have to worry about every machine also
> > > running the above separate process to write to Kafka.
> > >
> > > B) Work around the above limitations. Auto-discover of brokers is
> > terrible
> > > since we don't foresee us adding/removing brokers that frequently. The
> > lack
> > > of auto-discover of partitions is definitely a loss since we now have
> to
> > > know which broker/partition to read from at all times. Of course we can
> > > just write to Kafka using the above Gems and have our consumers written
> > in
> > > another language.
> > >
> > > C)?
> > >
> > > Any thoughts/opinions?
> > >
> > > Thanks
> > >
> > > - M
> > >
> > >
> >
>