Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka throw InvalidMessageException and lost data


Copy link to this message
-
Re: Kafka throw InvalidMessageException and lost data
Hi Neha,

I am sorry, we couldn't find the detail reason of this bug. But we do
believe that the bug is in java.nio package. Something bad may happen
during sync-producer sends data in byteBuffer or broker receives data from
the channel.

The serialize/deserialize process is right, because we enable valid-message
in producer, the crc checksum is right in client. But the crc checksum is
invalid in the server.
BTW: the bug is only reproduced in high concurrency situation.

Thanks.
On Mon, Mar 25, 2013 at 12:00 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> That's great. Would you know the details of the bug though ? It will be
> helpful for the community to understand.
>
> Thanks,
> Neha
>
> On Sunday, March 24, 2013, 王国栋 wrote:
>
> > Hi Neha & Jun,
> >
> > I think we have found the reason of this bug.
> > It is related to jdk versions. In the beginning, we use jdk1.6.20 to run
> > the test.
> >
> > Yesterday, we used jdk1.6.38. Everything is OK now.
> >
> > Thanks a lot for all your kind help and answer.
> >
> >
> >
> > On Thu, Mar 21, 2013 at 1:05 PM, Neha Narkhede <[EMAIL PROTECTED]
> <javascript:;>
> > >wrote:
> >
> > > Do you mind filing a bug and attaching the reproducible test case
> there ?
> > >
> > > Thanks,
> > > Neha
> > >
> > > On Wednesday, March 20, 2013, 王国栋 wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > We use one thread with one sync produce to send data to broker
> > > > (QPS:10k-15k, each log is about 1k bytes). The problem is reproduced.
> > > >
> > > > We have used Producer and SyncProducer in our test. The same
> Exception
> > > > appears.
> > > >
> > > > Thanks.
> > > >
> > > >
> > > >
> > > > On Thu, Mar 21, 2013 at 12:19 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > How many threads are you using?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Mar 20, 2013 at 7:33 PM, Yang Zhou <
> [EMAIL PROTECTED]
> > >
> > > > > wrote:
> > > > >
> > > > > > Sorry, I made a mistake, we use many threads producing at same
> > time.
> > > > > >
> > > > > >
> > > > > > 2013/3/20 Jun Rao <[EMAIL PROTECTED]>
> > > > > >
> > > > > > > How many producer instances do you have? Can you reproduce the
> > > > problem
> > > > > > with
> > > > > > > a single producer?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, Mar 20, 2013 at 12:29 AM, 王国栋 <[EMAIL PROTECTED]>
> > wrote:
> > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > we do not use any compression in our test.
> > > > > > > >
> > > > > > > > We deploy producer and broker in the same machine. The
> problem
> > > > still
> > > > > > > > exists. We use sync producer, and send one message at a
> time(no
> > > > batch
> > > > > > > now).
> > > > > > > > We find that when the qps reaches more than 40k, the
> exception
> > > > > appears.
> > > > > > > So
> > > > > > > > I don't think it's the underlying system error.
> > > > > > > >
> > > > > > > > Any suggestions if we want to do some debug on kafka
> > > > > > > > serialization/deserialization?
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Mar 20, 2013 at 12:10 AM, Jun Rao <[EMAIL PROTECTED]>
> > > > wrote:
> > > > > > > >
> > > > > > > > > It basically means that the broker is expecting to read
> > certain
> > > > > > number
> > > > > > > of
> > > > > > > > > bytes in a buffer received from socket, but there are fewer
> > > bytes
> > > > > > than
> > > > > > > > > expected in the buffer. Possible causes are (1) a bug in
> > Kafka
> > > > > > request
> > > > > > > > > serialization/deserialization logic; (2) corruption in the
> > > > > underlying
> > > > > > > > > system such as network.
> > > > > > > > >
> > > > > > > > > BTW, did you enable compression in your producer?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Mon, Mar 18, 2013 at 10
Guodong Wang
王国栋

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB