Hi, I am running the console demo comes with 0.7.2  in getting started guide. All is working fine. Then I use stdio redirect a file of 30K lines of messages into the producer and I received  an error "ERROR Event queue is full of unsent messages, could not send event:..." and exception stack shows:
Exception in thread "main" kafka.producer.async.QueueFullException: Event queue is full of unsent messages, could not send event:...

Is there a way in the produce.properties to configure the size of producer queue?   I know the producer is running in synchronous mode. How do I solve this problem or my usage is totally wrong?

Thanks
Jamie

To use async, set producer.type to async ; The default queue size is 10000;
and the default batch size is 200.
On Mon, Nov 26, 2012 at 2:28 PM, Jamie Wang <[EMAIL PROTECTED]> wrote:

> Hi, I am running the console demo comes with 0.7.2  in getting started
> guide. All is working fine. Then I use stdio redirect a file of 30K lines
> of messages into the producer and I received  an error "ERROR Event queue
> is full of unsent messages, could not send event:..." and exception stack
> shows:
> Exception in thread "main" kafka.producer.async.QueueFullException: Event
> queue is full of unsent messages, could not send event:...
>
> Is there a way in the produce.properties to configure the size of producer
> queue?   I know the producer is running in synchronous mode. How do I solve
> this problem or my usage is totally wrong?
>
> Thanks
> Jamie
>

This will happen if you push data to the producer at a higher rate
than it is able to send to the server.
queue.size allows you to configure the size of the producer queue
(defaults to 10K). Also,
queue.enqueueTimeout.ms, if set to -1 will lead to blocking behaviour
instead of the producer throwing
QueueFullExceptions.

Thanks,
Neha

On Mon, Nov 26, 2012 at 4:06 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> To use async, set producer.type to async ; The default queue size is 10000;
> and the default batch size is 200.
>
>
> On Mon, Nov 26, 2012 at 2:28 PM, Jamie Wang <[EMAIL PROTECTED]> wrote:
>
>> Hi, I am running the console demo comes with 0.7.2  in getting started
>> guide. All is working fine. Then I use stdio redirect a file of 30K lines
>> of messages into the producer and I received  an error "ERROR Event queue
>> is full of unsent messages, could not send event:..." and exception stack
>> shows:
>> Exception in thread "main" kafka.producer.async.QueueFullException: Event
>> queue is full of unsent messages, could not send event:...
>>
>> Is there a way in the produce.properties to configure the size of producer
>> queue?   I know the producer is running in synchronous mode. How do I solve
>> this problem or my usage is totally wrong?
>>
>> Thanks
>> Jamie
>>

I wonder if this makes sense for the default. I agree that most people who
are choosing async would prefer to drop the message rather than block since
it is async. But I wonder if we should default to -1 anyway and let them
set this. The reason is that I think benchmarking and other cases always
run into this and it matches the behavior of other messaging systems.

-Jay
On Mon, Nov 26, 2012 at 4:40 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> This will happen if you push data to the producer at a higher rate
> than it is able to send to the server.
> queue.size allows you to configure the size of the producer queue
> (defaults to 10K). Also,
> queue.enqueueTimeout.ms, if set to -1 will lead to blocking behaviour
> instead of the producer throwing
> QueueFullExceptions.
>
> Thanks,
> Neha
>
> On Mon, Nov 26, 2012 at 4:06 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> > To use async, set producer.type to async ; The default queue size is
> 10000;
> > and the default batch size is 200.
> >
> >
> > On Mon, Nov 26, 2012 at 2:28 PM, Jamie Wang <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi, I am running the console demo comes with 0.7.2  in getting started
> >> guide. All is working fine. Then I use stdio redirect a file of 30K
> lines
> >> of messages into the producer and I received  an error "ERROR Event
> queue
> >> is full of unsent messages, could not send event:..." and exception
> stack
> >> shows:
> >> Exception in thread "main" kafka.producer.async.QueueFullException:
> Event
> >> queue is full of unsent messages, could not send event:...
> >>
> >> Is there a way in the produce.properties to configure the size of
> producer
> >> queue?   I know the producer is running in synchronous mode. How do I
> solve
> >> this problem or my usage is totally wrong?
> >>
> >> Thanks
> >> Jamie
> >>
>