Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Kafka throw InvalidMessageException and lost data


+
Helin Xiang 2013-03-19, 03:22
+
Jun Rao 2013-03-19, 04:58
+
Helin Xiang 2013-03-19, 05:13
+
Neha Narkhede 2013-03-19, 05:55
+
王国栋 2013-03-20, 07:34
+
Jun Rao 2013-03-19, 16:10
+
王国栋 2013-03-20, 07:29
+
Jun Rao 2013-03-20, 14:29
+
Yang Zhou 2013-03-21, 02:09
+
Yang Zhou 2013-03-21, 02:33
+
Jun Rao 2013-03-21, 04:20
+
王国栋 2013-03-21, 04:43
+
Neha Narkhede 2013-03-21, 05:05
Copy link to this message
-
Re: Kafka throw InvalidMessageException and lost data
王国栋 2013-03-25, 03:58
Hi Neha & Jun,

I think we have found the reason of this bug.
It is related to jdk versions. In the beginning, we use jdk1.6.20 to run
the test.

Yesterday, we used jdk1.6.38. Everything is OK now.

Thanks a lot for all your kind help and answer.

On Thu, Mar 21, 2013 at 1:05 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Do you mind filing a bug and attaching the reproducible test case there ?
>
> Thanks,
> Neha
>
> On Wednesday, March 20, 2013, 王国栋 wrote:
>
> > Hi Jun,
> >
> > We use one thread with one sync produce to send data to broker
> > (QPS:10k-15k, each log is about 1k bytes). The problem is reproduced.
> >
> > We have used Producer and SyncProducer in our test. The same Exception
> > appears.
> >
> > Thanks.
> >
> >
> >
> > On Thu, Mar 21, 2013 at 12:19 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > How many threads are you using?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Mar 20, 2013 at 7:33 PM, Yang Zhou <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Sorry, I made a mistake, we use many threads producing at same time.
> > > >
> > > >
> > > > 2013/3/20 Jun Rao <[EMAIL PROTECTED]>
> > > >
> > > > > How many producer instances do you have? Can you reproduce the
> > problem
> > > > with
> > > > > a single producer?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Mar 20, 2013 at 12:29 AM, 王国栋 <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > we do not use any compression in our test.
> > > > > >
> > > > > > We deploy producer and broker in the same machine. The problem
> > still
> > > > > > exists. We use sync producer, and send one message at a time(no
> > batch
> > > > > now).
> > > > > > We find that when the qps reaches more than 40k, the exception
> > > appears.
> > > > > So
> > > > > > I don't think it's the underlying system error.
> > > > > >
> > > > > > Any suggestions if we want to do some debug on kafka
> > > > > > serialization/deserialization?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Mar 20, 2013 at 12:10 AM, Jun Rao <[EMAIL PROTECTED]>
> > wrote:
> > > > > >
> > > > > > > It basically means that the broker is expecting to read certain
> > > > number
> > > > > of
> > > > > > > bytes in a buffer received from socket, but there are fewer
> bytes
> > > > than
> > > > > > > expected in the buffer. Possible causes are (1) a bug in Kafka
> > > > request
> > > > > > > serialization/deserialization logic; (2) corruption in the
> > > underlying
> > > > > > > system such as network.
> > > > > > >
> > > > > > > BTW, did you enable compression in your producer?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Mon, Mar 18, 2013 at 10:12 PM, Helin Xiang <
> [EMAIL PROTECTED]
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > thanks Jun.
> > > > > > > >
> > > > > > > > we are using java producer.
> > > > > > > > does the last exception
> > > > > > > > "java.lang.IllegalArgumentException
> > > > > > > >     at java.nio.Buffer.limit(Buffer.java:266)
> > > > > > > > "
> > > > > > > > also means the broker received corrupted messages ?  sorry i
> am
> > > not
> > > > > > > > familiar with java nio.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 19, 2013 at 12:58 PM, Jun Rao <[EMAIL PROTECTED]>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hmm, both log4j messages suggest that the broker received
> > some
> > > > > > > corrupted
> > > > > > > > > produce requests. Are you using the java producer? Also, we
> > > have
> > > > > seen
> > > > > > > > that
> > > > > > > > > network router problems caused corrupted requests before.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Mon, Mar 18, 2013 at 8:22 PM, Helin Xiang <
> > --
> > Guodong Wang
> > 王国栋
> >
>

--
Guodong Wang
王国栋

 
+
Neha Narkhede 2013-03-25, 04:01
+
王国栋 2013-03-25, 04:23