-
Re: A "Java heap space" questionJun Rao 2012-12-28, 22:32
Then, compression won't help. Try increasing the heap size. If that doesn't
help, you may need to use more brokers.
Thanks,
Jun
On Thu, Dec 27, 2012 at 10:26 PM, xingcan <[EMAIL PROTECTED]> wrote:
> Jun,
>
> Out messages are not plain text. Most of them are JPEG files. I'm not sure
> if compression will be useful.
>
>
> 2012/12/28 Jun Rao <[EMAIL PROTECTED]>
>
> > Are messages compressed? If not, compressing them could save java heap
> > space.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Dec 27, 2012 at 6:11 PM, xingcan <[EMAIL PROTECTED]> wrote:
> >
> > > Jun,
> > > The peak sending rate is about 200/s(there are 3 SAS disks for 3
> > > partitions). We use sync producer for we can not tolerate message
> > > discarding. The truth is our messages' producing rate is increasing
> these
> > > days and the kafka had been running perfectly for a long time before.
> > I'll
> > > use a bigger heap size and monitor for a period. On such a producing
> rate
> > > and message size, is there any advices for config files(e.g socket
> > buffer)
> > > you can give me? Thanks.
> > >
> > >
> > >
> > > 2012/12/27 Jun Rao <[EMAIL PROTECTED]>
> > >
> > > > 400KB messages are relatively large. How many messages are you
> sending
> > > per
> > > > sec? How big is your JVM heap? You may need a bigger heap size.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Dec 26, 2012 at 5:41 PM, xingcan <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I run 'kafka-0.7.2' on a single node and today I got some error
> logs
> > > > below.
> > > > >
> > > > > ...
> > > > > [2012-12-26 08:51:18,496] INFO End registering broker topic
> > > > > /brokers/topics/jnits/1 (kafka.server.KafkaZooKeeper)
> > > > > 161557 [2012-12-26 08:51:19,103] INFO Closing socket connection to
> /
> > > > > 172.18.116.52. (kafka.network.Processor)
> > > > > 161558 [2012-12-26 08:51:18,416] ERROR OOME with size 313705
> > > > > (kafka.network.BoundedByteBufferReceive)
> > > > > 161559 java.lang.OutOfMemoryError: Java heap space
> > > > > 161560 [2012-12-26 08:51:18,341] ERROR OOME with size 317477
> > > > > (kafka.network.BoundedByteBufferReceive)
> > > > > 161561 java.lang.OutOfMemoryError: Java heap space
> > > > > 161562 [2012-12-26 08:51:19,178] ERROR Closing socket for
> > > > /172.18.0.34because
> > > > > of error (kafka.network.Processor)
> > > > > 161563 java.lang.OutOfMemoryError: Java heap space
> > > > > 161564 [2012-12-26 08:51:19,178] ERROR Closing socket for
> > > > > /172.18.116.46 because
> > > > > of error (kafka.network.Processor)
> > > > > 161565 java.lang.OutOfMemoryError: Java heap space
> > > > > 161566 [2012-12-26 08:51:19,375] ERROR OOME with size 269336
> > > > > (kafka.network.BoundedByteBufferReceive)
> > > > > 161567 java.lang.OutOfMemoryError: Java heap space
> > > > > 161568 [2012-12-26 08:51:19,375] ERROR Closing socket for
> > > > > /172.18.113.38 because
> > > > > of error (kafka.network.Processor)
> > > > > 161569 java.lang.OutOfMemoryError: Java heap space
> > > > > ...
> > > > >
> > > > > The size of each message is about 400KB. Maybe the messages are
> > > produced
> > > > > too fast or the disk can not be written at such a rate(about
> > > > 33MB/s/disk)?
> > > > > Can anyone help me explain it? Thanks a lot.
> > > > > --
> > > > > *Xingcan*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Xingcan*
> > >
> >
>
>
>
> --
> *Xingcan*
>
help, you may need to use more brokers.
Thanks,
Jun
On Thu, Dec 27, 2012 at 10:26 PM, xingcan <[EMAIL PROTECTED]> wrote:
> Jun,
>
> Out messages are not plain text. Most of them are JPEG files. I'm not sure
> if compression will be useful.
>
>
> 2012/12/28 Jun Rao <[EMAIL PROTECTED]>
>
> > Are messages compressed? If not, compressing them could save java heap
> > space.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Dec 27, 2012 at 6:11 PM, xingcan <[EMAIL PROTECTED]> wrote:
> >
> > > Jun,
> > > The peak sending rate is about 200/s(there are 3 SAS disks for 3
> > > partitions). We use sync producer for we can not tolerate message
> > > discarding. The truth is our messages' producing rate is increasing
> these
> > > days and the kafka had been running perfectly for a long time before.
> > I'll
> > > use a bigger heap size and monitor for a period. On such a producing
> rate
> > > and message size, is there any advices for config files(e.g socket
> > buffer)
> > > you can give me? Thanks.
> > >
> > >
> > >
> > > 2012/12/27 Jun Rao <[EMAIL PROTECTED]>
> > >
> > > > 400KB messages are relatively large. How many messages are you
> sending
> > > per
> > > > sec? How big is your JVM heap? You may need a bigger heap size.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Dec 26, 2012 at 5:41 PM, xingcan <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I run 'kafka-0.7.2' on a single node and today I got some error
> logs
> > > > below.
> > > > >
> > > > > ...
> > > > > [2012-12-26 08:51:18,496] INFO End registering broker topic
> > > > > /brokers/topics/jnits/1 (kafka.server.KafkaZooKeeper)
> > > > > 161557 [2012-12-26 08:51:19,103] INFO Closing socket connection to
> /
> > > > > 172.18.116.52. (kafka.network.Processor)
> > > > > 161558 [2012-12-26 08:51:18,416] ERROR OOME with size 313705
> > > > > (kafka.network.BoundedByteBufferReceive)
> > > > > 161559 java.lang.OutOfMemoryError: Java heap space
> > > > > 161560 [2012-12-26 08:51:18,341] ERROR OOME with size 317477
> > > > > (kafka.network.BoundedByteBufferReceive)
> > > > > 161561 java.lang.OutOfMemoryError: Java heap space
> > > > > 161562 [2012-12-26 08:51:19,178] ERROR Closing socket for
> > > > /172.18.0.34because
> > > > > of error (kafka.network.Processor)
> > > > > 161563 java.lang.OutOfMemoryError: Java heap space
> > > > > 161564 [2012-12-26 08:51:19,178] ERROR Closing socket for
> > > > > /172.18.116.46 because
> > > > > of error (kafka.network.Processor)
> > > > > 161565 java.lang.OutOfMemoryError: Java heap space
> > > > > 161566 [2012-12-26 08:51:19,375] ERROR OOME with size 269336
> > > > > (kafka.network.BoundedByteBufferReceive)
> > > > > 161567 java.lang.OutOfMemoryError: Java heap space
> > > > > 161568 [2012-12-26 08:51:19,375] ERROR Closing socket for
> > > > > /172.18.113.38 because
> > > > > of error (kafka.network.Processor)
> > > > > 161569 java.lang.OutOfMemoryError: Java heap space
> > > > > ...
> > > > >
> > > > > The size of each message is about 400KB. Maybe the messages are
> > > produced
> > > > > too fast or the disk can not be written at such a rate(about
> > > > 33MB/s/disk)?
> > > > > Can anyone help me explain it? Thanks a lot.
> > > > > --
> > > > > *Xingcan*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Xingcan*
> > >
> >
>
>
>
> --
> *Xingcan*
>
