Hi

I have a question about kafka servers.
Can I generate several consumer thread when I start kafka servers?
In other words, right now in our program kafka servers and consumers are
two different processes. We use SimpleConsumer instance and do
inter-process communication in order to fetch messages. We want to optimize
our system architecture through integrating consumers into kafka servers.
In this way, consumers will become kafka server threads and we can read
messages from shared memory. This will be more efficient than current
design.

Are there any api that allow me to do this or I can pass some flags to
kafka server to invoke this functionality?

--
Regards,
Sining Ma

 

Currently, we don't support embedded consumers in the broker. Doing that
requires running user-level code in the broker, which we haven't thought
about.

Are you concerned about network latency? That's probably in the order of
1ms. In terms of throughput, if you make fetch size big enough, you can
achieve good throughput even with a remote consumer, since the network
overhead is amortized.

Thanks,

Jun

On Fri, Mar 1, 2013 at 3:05 PM, Sining Ma <[EMAIL PROTECTED]> wrote:

> Hi
>
> I have a question about kafka servers.
> Can I generate several consumer thread when I start kafka servers?
> In other words, right now in our program kafka servers and consumers are
> two different processes. We use SimpleConsumer instance and do
> inter-process communication in order to fetch messages. We want to optimize
> our system architecture through integrating consumers into kafka servers.
> In this way, consumers will become kafka server threads and we can read
> messages from shared memory. This will be more efficient than current
> design.
>
> Are there any api that allow me to do this or I can pass some flags to
> kafka server to invoke this functionality?
>
> --
> Regards,
> Sining Ma
>