I am looking at the reliability aspects of the system. I want to have some
clarity on:

1. Producer side spooling: If none of the broker is reachable by producer,
does the producer spool to disk and retry. I could not locate the code for
it. Can anybody point me the code if it does spooling ?

2. How folks integrate with the producers ? Do they directly stream messages
from the *embedded* client from producers or run a local agent which
forwards the messages to the broker ?

3. If a message has to be given to a particular partition (based on the
partitioning function) and the broker hosting that partition is down, then
what happen to that message. It can't be given to any other partition as it
will void the partitioning contract. Right ?
Thanks
Sharad Agarwal
Hadoop and Avro Committer
Technology Platforms, InMobi
Disclaimer: Opinions expressed are my own and do not represent past or
present employers.

Sarad,

Good questions.

1. There is currently no producer side spooling. If all brokers are down,
messages are dropped.

2. In most cases, your client will embed the producer to send messages to
the broker.

3. Currently, if you are doing semantic partitioning and a broker is down,
we simply shrink the # of available partitions and reapply the partitioning
function accordingly. This means that a key could be mapped to a different
partition. This is definitely a problem for some applications. We hope
replication will solve this work.

Thanks,

Jun

On Thu, Oct 13, 2011 at 10:58 PM, Sharad Agarwal <[EMAIL PROTECTED]>wrote:

> I am looking at the reliability aspects of the system. I want to have some
> clarity on:
>
> 1. Producer side spooling: If none of the broker is reachable by producer,
> does the producer spool to disk and retry. I could not locate the code for
> it. Can anybody point me the code if it does spooling ?
>
> 2. How folks integrate with the producers ? Do they directly stream
> messages
> from the *embedded* client from producers or run a local agent which
> forwards the messages to the broker ?
>
> 3. If a message has to be given to a particular partition (based on the
> partitioning function) and the broker hosting that partition is down, then
> what happen to that message. It can't be given to any other partition as it
> will void the partitioning contract. Right ?
>
>
> Thanks
> Sharad Agarwal
> Hadoop and Avro Committer
> Technology Platforms, InMobi
> Disclaimer: Opinions expressed are my own and do not represent past or
> present employers.
>

Thanks. Filed:

https://issues.apache.org/jira/browse/KAFKA-156
https://issues.apache.org/jira/browse/KAFKA-157
--
Thanks
Sharad Agarwal
Hadoop and Avro Committer
Technology Platforms, InMobi
*Disclaimer: Opinions expressed here are my own and do not represent past or
present employers.*