Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Guarantees


+
graham sanderson 2012-07-13, 02:08
+
Jun Rao 2012-07-14, 18:32
+
Neha Narkhede 2012-07-20, 23:00
Thanks Neha, I really meant, yes I may lose some messages in the meanwhile, but should I expect new messages after everything gets back to normal to be delivered (unless my code throws an exception and kills a worker thread, which it wasn't)

On Jul 20, 2012, at 6:00 PM, Neha Narkhede wrote:

> Graham,
>
> It really depends on what sort of network outage. The producer, whether zk
> or not, can be configured to retry couple of times. If it runs out of
> retries during this outage, it will drop the messages and they will be
> lost.
>
> Thanks,
> Neha
>
> On Sat, Jul 14, 2012 at 11:32 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> The pipeline is supposed to recover from the network outage. There could be
>> bugs, especially in the ZK-based producer since it's relatively new.
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Jul 12, 2012 at 7:08 PM, graham sanderson <[EMAIL PROTECTED]> wrote:
>>
>>> Hi, so I happened to be going to demo a prototype built with kafka in a
>>> borrowed large room which I discovered had insufficient/flaky wireless.
>> Was
>>> using zookeeper config, and getting lots of timeouts etc. Since this was
>>> the first time I had used kafka and I hadn't done any off path testing,
>> my
>>> first course of action was to find a hard wire, which I did and all the
>>> timeouts disappeared. The demo was great. Note that even with the flaky
>>> wireless, messages generally still seemed to be getting delivered, but
>> not
>>> always as far as I could tell (or perhaps with high latency - was more
>>> focused on having a working demo than debugging)
>>>
>>> I'm using 0.7 atm, though I'm not sure if that matters.
>>>
>>> My somewhat question is, given a simple scenario using kafka/zookeeper
>>> (prior to all the exciting fault tolerance work going on right now):
>>>
>>> 1) Lets say I have zookeeper server, kafka server, producer, and consumer
>>> running on a perfect network. And I successfully send a message from
>>> producer to consumer
>>> 2) All JVMs stay up, however I lose network connectivity between some or
>>> all of them for some time
>>> 3) The network becomes perfect again.
>>> 4) I wait for some time for everyone to reconnect/re-negociate to their
>>> best ability
>>>
>>> Following that, should I expect a new message from the producer to reach
>>> the consumer, or can the system get into a broken state?… I swear I saw
>>> such a message not delivered, but I can't say for sure… I can certainly
>>> investigate further by trying to reproduce again and wading thru the many
>>> logged errors, but if someone already knows the answer that'd be awesome!
>>>
>>> Thanks,
>>>
>>> Graham.
>>>
>>>
>>

+
Neha Narkhede 2012-07-21, 00:53
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB