Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka throw InvalidMessageException and lost data


Copy link to this message
-
Re: Kafka throw InvalidMessageException and lost data
王国栋 2013-03-25, 04:23
Hi Neha,

I am sorry, we couldn't find the detail reason of this bug. But we do
believe that the bug is in java.nio package. Something bad may happen
during sync-producer sends data in byteBuffer or broker receives data from
the channel.

The serialize/deserialize process is right, because we enable valid-message
in producer, the crc checksum is right in client. But the crc checksum is
invalid in the server.
BTW: the bug is only reproduced in high concurrency situation.

Thanks.
On Mon, Mar 25, 2013 at 12:00 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> That's great. Would you know the details of the bug though ? It will be
> helpful for the community to understand.
>
> Thanks,
> Neha
>
> On Sunday, March 24, 2013, 王国栋 wrote:
>
> > Hi Neha & Jun,
> >
> > I think we have found the reason of this bug.
> > It is related to jdk versions. In the beginning, we use jdk1.6.20 to run
> > the test.
> >
> > Yesterday, we used jdk1.6.38. Everything is OK now.
> >
> > Thanks a lot for all your kind help and answer.
> >
> >
> >
> > On Thu, Mar 21, 2013 at 1:05 PM, Neha Narkhede <[EMAIL PROTECTED]
> <javascript:;>
> > >wrote:
> >
> > > Do you mind filing a bug and attaching the reproducible test case
> there ?
> > >
> > > Thanks,
> > > Neha
> > >
> > > On Wednesday, March 20, 2013, 王国栋 wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > We use one thread with one sync produce to send data to broker
> > > > (QPS:10k-15k, each log is about 1k bytes). The problem is reproduced.
> > > >
> > > > We have used Producer and SyncProducer in our test. The same
> Exception
> > > > appears.
> > > >
> > > > Thanks.
> > > >
> > > >
> > > >
> > > > On Thu, Mar 21, 2013 at 12:19 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > How many threads are you using?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Mar 20, 2013 at 7:33 PM, Yang Zhou <
> [EMAIL PROTECTED]
> > >
> > > > > wrote:
> > > > >
> > > > > > Sorry, I made a mistake, we use many threads producing at same
> > time.
> > > > > >
> > > > > >
> > > > > > 2013/3/20 Jun Rao <[EMAIL PROTECTED]>
> > > > > >
> > > > > > > How many producer instances do you have? Can you reproduce the
> > > > problem
> > > > > > with
> > > > > > > a single producer?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, Mar 20, 2013 at 12:29 AM, 王国栋 <[EMAIL PROTECTED]>
> > wrote:
> > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > we do not use any compression in our test.
> > > > > > > >
> > > > > > > > We deploy producer and broker in the same machine. The
> problem
> > > > still
> > > > > > > > exists. We use sync producer, and send one message at a
> time(no
> > > > batch
> > > > > > > now).
> > > > > > > > We find that when the qps reaches more than 40k, the
> exception
> > > > > appears.
> > > > > > > So
> > > > > > > > I don't think it's the underlying system error.
> > > > > > > >
> > > > > > > > Any suggestions if we want to do some debug on kafka
> > > > > > > > serialization/deserialization?
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Mar 20, 2013 at 12:10 AM, Jun Rao <[EMAIL PROTECTED]>
> > > > wrote:
> > > > > > > >
> > > > > > > > > It basically means that the broker is expecting to read
> > certain
> > > > > > number
> > > > > > > of
> > > > > > > > > bytes in a buffer received from socket, but there are fewer
> > > bytes
> > > > > > than
> > > > > > > > > expected in the buffer. Possible causes are (1) a bug in
> > Kafka
> > > > > > request
> > > > > > > > > serialization/deserialization logic; (2) corruption in the
> > > > > underlying
> > > > > > > > > system such as network.
> > > > > > > > >
> > > > > > > > > BTW, did you enable compression in your producer?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Mon, Mar 18, 2013 at 10
Guodong Wang
王国栋