Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Re: Loading kafka server within a Java process


+
Jay Kreps 2013-02-14, 22:40
+
Eric Tschetter 2013-02-14, 22:44
Copy link to this message
-
Re: Loading kafka server within a Java process
Consumer + broker embedding can be done, I believe.  It's just
producer+broker embedding that they don't like :).

--Eric
On Thu, Feb 14, 2013 at 4:51 PM, Jamie Wang <[EMAIL PROTECTED]> wrote:

> Thank you both for the info. I'll keep my eye open since I do have a need
> to embed a consumer in the same process.
>
> Jamie
>
> -----Original Message-----
> From: Eric Tschetter [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, February 14, 2013 2:44 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Loading kafka server within a Java process
>
> Jamie,
>
> As Jay says, this is definitely possible.  We also do it at Metamarkets.
>
> The one thing that we found odd when doing it though is that you cannot (or
> couldn't with the version we are using, I'm not sure about 0.8) embed a
> producer that produces for an embedded broker.  You instead have to have
> your producer talk over the "network" to the broker even though it is
> running inside the same process.  We haven't run into any issues doing this
> yet, but it's something to be aware of.
>
> --Eric
>
>
> On Thu, Feb 14, 2013 at 4:39 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
> > Yes, it will work. We do this at LinkedIn, actually.
> >
> > -Jay
> >
> > On Thu, Feb 14, 2013 at 12:15 PM, Jamie Wang <[EMAIL PROTECTED]>
> > wrote:
> > > I have a situation where we have constraint on the number of processes
> > we can use and hence, is it possible to load kafka server within another
> > Java process? Looking at the kafka.kafka it seems that I can create a
> Java
> > object that does most of what Kafka.kafka.main() do.  Any thoughts if
> this
> > will work? What are some of the things I need to watch out? And is there
> > another better way to do this?  Thank you for your time.
> > >
> > > Jamie
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB