Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Re: Batch size in producer


+
Neha Narkhede 2012-12-20, 22:54
Copy link to this message
-
Re: Batch size in producer
You can call the close() API on the producer

Thanks,
Neha
On Thu, Dec 20, 2012 at 3:06 PM, Subhash Agrawal <[EMAIL PROTECTED]>wrote:

> Thanks Neha.
>
> How can I shutdown the producer? is it ctrl-c?
>
> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, December 20, 2012 2:54 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Batch size in producer
>
> > How big should be the batch size?
> >
>
> In production at Linkedin, we use a batch size of 200, which has worked
> pretty well.
>
>
> > What happens if producer client crashes before batch is full and messages
> > were still sitting in producer queue?  Does it recover those messages
> when
> > we restart the producer?
> >
>
> If the producer shuts down, it flushes out the messages in the internal
> queue. If it crashes, it does not get a chance to do that and there is no
> good alternative to this.
>
>
> > Are these messages stored in memory or disk?
> >
>
> In memory.
>
> Thanks,
> Neha
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB