Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> question about broker


Copy link to this message
-
Re: question about broker
This is pretty fundamental to the contract the the broker provides.
Without changing code you could use some LD_PRELOAD hackery to disable
fsync and then set log.flush.interval=1 to one.  But that's really not
recommended.

On 06/20/2012 01:11 PM, R S wrote:
> Thank you Neha ,
>
> Just to clarify , when you say high watermark , my understanding of that is
> the maximum offset which is available to consumer .
> The reason iam asking about this is , we want to use kafka for ultra fast
> messaging to do budget updates in shortest possible time , hence we just
> want to send messages to our aggregation service as fast as possible so was
> curious if there is a way we can just skip write to the disk and  just
> support Pub-Sub.
>
> regards,
> rks
>
> On Wed, Jun 20, 2012 at 10:34 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> The broker maintains a high watermark for each topic partition log.
>> This high watermark moves only after the messages are written to disk.
>> And, the consumer can only consume data exposed by this high
>> watermark.
>>
>> Thanks,
>> Neha
>>
>> On Wed, Jun 20, 2012 at 10:02 AM, R S <[EMAIL PROTECTED]> wrote:
>>> Hi,
>>>
>>> When producer sends messages to broker on a particular topic and at the
>>> sametime  suppose consumer is fetching messages from the same topic. Will
>>> broker be sending message to consumer only after writing the message to
>>> disk or would writing to disk will happen in a asynchronous way.
>>>
>>>
>>> regards,
>>> rks
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB