Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - generate message consumers as kafka server threads


+
Sining Ma 2013-03-01, 23:05
Copy link to this message
-
Re: generate message consumers as kafka server threads
Jun Rao 2013-03-02, 16:06
Currently, we don't support embedded consumers in the broker. Doing that
requires running user-level code in the broker, which we haven't thought
about.

Are you concerned about network latency? That's probably in the order of
1ms. In terms of throughput, if you make fetch size big enough, you can
achieve good throughput even with a remote consumer, since the network
overhead is amortized.

Thanks,

Jun

On Fri, Mar 1, 2013 at 3:05 PM, Sining Ma <[EMAIL PROTECTED]> wrote:

> Hi
>
> I have a question about kafka servers.
> Can I generate several consumer thread when I start kafka servers?
> In other words, right now in our program kafka servers and consumers are
> two different processes. We use SimpleConsumer instance and do
> inter-process communication in order to fetch messages. We want to optimize
> our system architecture through integrating consumers into kafka servers.
> In this way, consumers will become kafka server threads and we can read
> messages from shared memory. This will be more efficient than current
> design.
>
> Are there any api that allow me to do this or I can pass some flags to
> kafka server to invoke this functionality?
>
> --
> Regards,
> Sining Ma
>