Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - Kafka/Hadoop consumers and producers


Copy link to this message
-
Re: Kafka/Hadoop consumers and producers
Andrew Otto 2013-08-13, 23:17
>  may merge a bit of your work into bigtop-989 if that's ok with you.
Merge away!  Happy to help. :)

> I'll ask on bigtop regarding the .deb requirement - it seems they don't abide by this.

Yeah, there seems to be a constant struggle between the 'java way' of doing things, e.g. Maven downloading the internet, and the 'debian way', e.g. be paranoid about everything, make sure the build process is 100% repeatable.

Bigtop should definitely do whatever Bigtop thinks is best.  This Makefile technique works for us now, but probably will require a lot of manual maintenance as Kafka grows.
On Aug 13, 2013, at 6:03 PM, Kam Kasravi <[EMAIL PROTECTED]> wrote:

> Thanks - I'll ask on bigtop regarding the .deb requirement - it seems they don't abide by this.
> I may merge a bit of your work into bigtop-989 if that's ok with you. I do know the bigtop folks
> would like to see sbt support.
>
> From: Andrew Otto <[EMAIL PROTECTED]>
> To: Kam Kasravi <[EMAIL PROTECTED]>
> Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Ken Goodhope <[EMAIL PROTECTED]>; Andrew Psaltis <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Felix GV <[EMAIL PROTECTED]>; Cosmin Lehene <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Tuesday, August 13, 2013 1:03 PM
> Subject: Re: Kafka/Hadoop consumers and producers
>
> > What installs all the kafka dependencies under /usr/share/java?
>
>
> The debian/ work was done mostly by another WMF staffer.  We tried and tried to make sbt behave with debian standards, most importantly the one that requires that .debs can be created without needing to connect to the internet, aside from official apt repositories.
>
> Many of the /usr/share/java dependencies are handled by apt.  Any that aren't available in an official apt somewhere have been manually added to the ext/ directory.
>
> The sbt build system has been replaced with Make:
> https://github.com/wikimedia/operations-debs-kafka/blob/debian/debian/patches/our-own-build-system.patch
>
> You should be able to build a .deb by checking out the debian branch and running:
>
>   git-buildpackage -uc -us
>
> -Ao
>
>
>
>
>
>
> On Aug 13, 2013, at 1:34 PM, Kam Kasravi <[EMAIL PROTECTED]> wrote:
>
> > Thanks Andrew - I like the shell wrapper - very clean and simple.
> > What installs all the kafka dependencies under /usr/share/java?
> >
> > From: Andrew Otto <[EMAIL PROTECTED]>
> > To: Kam Kasravi <[EMAIL PROTECTED]>
> > Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Ken Goodhope <[EMAIL PROTECTED]>; Andrew Psaltis <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Felix GV <[EMAIL PROTECTED]>; Cosmin Lehene <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> > Sent: Monday, August 12, 2013 7:00 PM
> > Subject: Re: Kafka/Hadoop consumers and producers
> >
> > We've done a bit of work over at Wikimedia to debianize Kafka and make it behave like a regular service.
> >
> > https://github.com/wikimedia/operations-debs-kafka/blob/debian/debian
> >
> > Most relevant, Ken, is an init script for Kafka:
> >  https://github.com/wikimedia/operations-debs-kafka/blob/debian/debian/kafka.init
> >
> > And a bin/kafka shell wrapper for the kafka/bin/*.sh scripts:
> >  https://github.com/wikimedia/operations-debs-kafka/blob/debian/debian/bin/kafka
> >
> > I'm about to add an init script for MirrorMaker as well, so mirroring can be demonized and run as a service.
> >
> >
> > On Aug 12, 2013, at 8:16 PM, Kam Kasravi <[EMAIL PROTECTED]> wrote:
> >
> > > I would like to do this refactoring since I did a high level consumer a while ago.
> > > A few weeks ago I had opened KAFKA-949 Kafka on Yarn which I was also hoping to add to contribute.