Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Batch size in producer


Copy link to this message
-
Re: Batch size in producer
> How big should be the batch size?
>

In production at Linkedin, we use a batch size of 200, which has worked
pretty well.
> What happens if producer client crashes before batch is full and messages
> were still sitting in producer queue?  Does it recover those messages when
> we restart the producer?
>

If the producer shuts down, it flushes out the messages in the internal
queue. If it crashes, it does not get a chance to do that and there is no
good alternative to this.
> Are these messages stored in memory or disk?
>

In memory.

Thanks,
Neha

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB