Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Loading kafka server within a Java process


Copy link to this message
-
Re: Loading kafka server within a Java process
Consumer + broker embedding can be done, I believe.  It's just
producer+broker embedding that they don't like :).

--Eric
On Thu, Feb 14, 2013 at 4:51 PM, Jamie Wang <[EMAIL PROTECTED]> wrote:

> Thank you both for the info. I'll keep my eye open since I do have a need
> to embed a consumer in the same process.
>
> Jamie
>
> -----Original Message-----
> From: Eric Tschetter [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, February 14, 2013 2:44 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Loading kafka server within a Java process
>
> Jamie,
>
> As Jay says, this is definitely possible.  We also do it at Metamarkets.
>
> The one thing that we found odd when doing it though is that you cannot (or
> couldn't with the version we are using, I'm not sure about 0.8) embed a
> producer that produces for an embedded broker.  You instead have to have
> your producer talk over the "network" to the broker even though it is
> running inside the same process.  We haven't run into any issues doing this
> yet, but it's something to be aware of.
>
> --Eric
>
>
> On Thu, Feb 14, 2013 at 4:39 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
> > Yes, it will work. We do this at LinkedIn, actually.
> >
> > -Jay
> >
> > On Thu, Feb 14, 2013 at 12:15 PM, Jamie Wang <[EMAIL PROTECTED]>
> > wrote:
> > > I have a situation where we have constraint on the number of processes
> > we can use and hence, is it possible to load kafka server within another
> > Java process? Looking at the kafka.kafka it seems that I can create a
> Java
> > object that does most of what Kafka.kafka.main() do.  Any thoughts if
> this
> > will work? What are some of the things I need to watch out? And is there
> > another better way to do this?  Thank you for your time.
> > >
> > > Jamie
> >
>