Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Kafka server start from resource negotiator(s)


Copy link to this message
-
Re: Kafka server start from resource negotiator(s)
Hi, Joe,

Thanks for the update. This looks great. A couple of questions and comments.

1. How general is the framework? For example, if I want to add the support
for Apache Yarn, is anything there can be reused?

2. My current feeling is that it's probably better to maintain those
container supports outside Kafka's git repo (e.g., can be on github).
People may be interested in integrating Kafka with Mesos, Yarn, VMs on EC2,
Azure, and Rackspace, etc. Maintaining all those in contrib may not be a
scalable solution.

Thanks,

Jun
On Sat, Jan 18, 2014 at 5:48 AM, Joe Stein <[EMAIL PROTECTED]> wrote:

> I got through some good research and testing this week on getting Kafka to
> run on Mesos.
>
> I updated the Mesos ticket
> https://issues.apache.org/jira/browse/KAFKA-1207with some thoughts.
>
> Next step I am going to draw out some diagrams and update the wiki page.  I
> was thinking of writing this in the contrib folder and adding two bin
> scripts if folks are interested in having this as part of the overall
> project.  There are few minor changes (like what I put in 1206) that could
> occur but for now I can get it working on 0.8.0 I believe (refactoring
> afterwards some changes non resource negotiator specific).
>
> For now I am going to work on it here
> https://github.com/stealthly/kafka-mesos (nothing there just yet) and once
> parts of it are working correctly upload patches to JIRA.
>
> Thoughts? Suggestions? Feedback? Tomatoes?
>
>
> /*******************************************
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
>
> On Thu, Jan 16, 2014 at 12:43 AM, Joe Stein <[EMAIL PROTECTED]> wrote:
>
> > Right now I am testing it that way. Once it is working and if that
> becomes
> > a problem then moving it into a mesosFramework project is no problem.
> if
> > we do that it would be beneficial if they were organized under some
> > resource manager folder or something.
> >
> > The way I wrote https://issues.apache.org/jira/browse/KAFKA-1206 was
> with
> > that in mind so folks have a light interface for doing this.  The
> specific
> > systems resource negotiator implementation could go in their project and
> > just importing the ResourceNegotiator trait in the project's reference of
> > the jar.
> >
> > /*******************************************
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
> >
> > On Thu, Jan 16, 2014 at 12:23 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> Hi, Joe,
> >>
> >> Thanks for looking into this. Do you plan to add a dependency to Mesos
> in
> >> kafka jar?
> >>
> >> Jun
> >>
> >>
> >> On Wed, Jan 15, 2014 at 10:50 AM, Joe Stein <[EMAIL PROTECTED]>
> wrote:
> >>
> >> > Hi, I am working at adding some code to the kafka.Kafka.scala file.
> >> >
> >> > What I was thinking for the Utils.loadProps is to create some
> >> > ResourceNegotiatedConfiguration class or something to sit with it so
> >> > various frameworks can be supported.
> >> >
> >> > For Mesos I am thinking of storing the properties in zookeeper. For a
> >> new
> >> > node joining => registering a default config (originally set by a
> tool)
> >> and
> >> > saving it unique for that broker in a znode.  This znode would be the
> >> input
> >> > to the serverConfig for the KafkaServerStartable.
> >> >
> >> > I am going to create a kafka.tools.ResourceNegotiator.ApacheMesos tool
> >> too.
> >> >  that tool will take in a server.properties file to set the original
> >> > default. Once loaded in zookeeper another command to take a property
> and
> >> > flag it with a function to run (like the "use new broker id value")
> and
> >> > such.
> >> >
> >> > I would rather have some implementation in Kafka.scala and have object