Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Kafka server start from resource negotiator(s)


Copy link to this message
-
Re: Kafka server start from resource negotiator(s)
Hi Jun, see inline

On Tue, Jan 21, 2014 at 1:14 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Hi, Joe,
>
> Thanks for the update. This looks great. A couple of questions and
> comments.
>
> 1. How general is the framework? For example, if I want to add the support
> for Apache Yarn, is anything there can be reused?
>

I don't think there is going to be much to share but I am honestly not far
enough along to know for sure.  I opened KAFKA-1206 for any (if any)
changes that they would both need or use to make it work from the existing
code (so we can launch brokers, producers and consumers in an abstract way
not just from shell scripts but from the resource negotiator system.  Quite
possible no change required...
>
> 2. My current feeling is that it's probably better to maintain those
> container supports outside Kafka's git repo (e.g., can be on github).
> People may be interested in integrating Kafka with Mesos, Yarn, VMs on EC2,
> Azure, and Rackspace, etc. Maintaining all those in contrib may not be a
> scalable solution.
>

I am neutral either way on this at the moment.  Right now, I am just
thinking about getting it working :)
If it lives only @ https://github.com/stealthly/kafka-mesos that is not a
bad thing (like how we do client libraries).
>
> Thanks,
>
> Jun
>
>
> On Sat, Jan 18, 2014 at 5:48 AM, Joe Stein <[EMAIL PROTECTED]> wrote:
>
> > I got through some good research and testing this week on getting Kafka
> to
> > run on Mesos.
> >
> > I updated the Mesos ticket
> > https://issues.apache.org/jira/browse/KAFKA-1207with some thoughts.
> >
> > Next step I am going to draw out some diagrams and update the wiki page.
>  I
> > was thinking of writing this in the contrib folder and adding two bin
> > scripts if folks are interested in having this as part of the overall
> > project.  There are few minor changes (like what I put in 1206) that
> could
> > occur but for now I can get it working on 0.8.0 I believe (refactoring
> > afterwards some changes non resource negotiator specific).
> >
> > For now I am going to work on it here
> > https://github.com/stealthly/kafka-mesos (nothing there just yet) and
> once
> > parts of it are working correctly upload patches to JIRA.
> >
> > Thoughts? Suggestions? Feedback? Tomatoes?
> >
> >
> > /*******************************************
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
> >
> > On Thu, Jan 16, 2014 at 12:43 AM, Joe Stein <[EMAIL PROTECTED]>
> wrote:
> >
> > > Right now I am testing it that way. Once it is working and if that
> > becomes
> > > a problem then moving it into a mesosFramework project is no problem.
> > if
> > > we do that it would be beneficial if they were organized under some
> > > resource manager folder or something.
> > >
> > > The way I wrote https://issues.apache.org/jira/browse/KAFKA-1206 was
> > with
> > > that in mind so folks have a light interface for doing this.  The
> > specific
> > > systems resource negotiator implementation could go in their project
> and
> > > just importing the ResourceNegotiator trait in the project's reference
> of
> > > the jar.
> > >
> > > /*******************************************
> > >  Joe Stein
> > >  Founder, Principal Consultant
> > >  Big Data Open Source Security LLC
> > >  http://www.stealth.ly
> > >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > ********************************************/
> > >
> > >
> > > On Thu, Jan 16, 2014 at 12:23 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > >> Hi, Joe,
> > >>
> > >> Thanks for looking into this. Do you plan to add a dependency to Mesos
> > in
> > >> kafka jar?
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Wed, Jan 15, 2014 at 10:50 AM, Joe Stein <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >> > Hi, I am working at adding some code to the kafka.Kafka.scala file.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB