Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> producer queue size


Copy link to this message
-
Re: producer queue size
I wonder if this makes sense for the default. I agree that most people who
are choosing async would prefer to drop the message rather than block since
it is async. But I wonder if we should default to -1 anyway and let them
set this. The reason is that I think benchmarking and other cases always
run into this and it matches the behavior of other messaging systems.

-Jay
On Mon, Nov 26, 2012 at 4:40 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> This will happen if you push data to the producer at a higher rate
> than it is able to send to the server.
> queue.size allows you to configure the size of the producer queue
> (defaults to 10K). Also,
> queue.enqueueTimeout.ms, if set to -1 will lead to blocking behaviour
> instead of the producer throwing
> QueueFullExceptions.
>
> Thanks,
> Neha
>
> On Mon, Nov 26, 2012 at 4:06 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> > To use async, set producer.type to async ; The default queue size is
> 10000;
> > and the default batch size is 200.
> >
> >
> > On Mon, Nov 26, 2012 at 2:28 PM, Jamie Wang <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi, I am running the console demo comes with 0.7.2  in getting started
> >> guide. All is working fine. Then I use stdio redirect a file of 30K
> lines
> >> of messages into the producer and I received  an error "ERROR Event
> queue
> >> is full of unsent messages, could not send event:..." and exception
> stack
> >> shows:
> >> Exception in thread "main" kafka.producer.async.QueueFullException:
> Event
> >> queue is full of unsent messages, could not send event:...
> >>
> >> Is there a way in the produce.properties to configure the size of
> producer
> >> queue?   I know the producer is running in synchronous mode. How do I
> solve
> >> this problem or my usage is totally wrong?
> >>
> >> Thanks
> >> Jamie
> >>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB