Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Kafka throw InvalidMessageException and lost data


+
Helin Xiang 2013-03-19, 03:22
+
Jun Rao 2013-03-19, 04:58
+
Helin Xiang 2013-03-19, 05:13
+
Neha Narkhede 2013-03-19, 05:55
+
王国栋 2013-03-20, 07:34
+
Jun Rao 2013-03-19, 16:10
+
王国栋 2013-03-20, 07:29
+
Jun Rao 2013-03-20, 14:29
+
Yang Zhou 2013-03-21, 02:09
+
Yang Zhou 2013-03-21, 02:33
+
Jun Rao 2013-03-21, 04:20
+
王国栋 2013-03-21, 04:43
Copy link to this message
-
Re: Kafka throw InvalidMessageException and lost data
Neha Narkhede 2013-03-21, 05:05
Do you mind filing a bug and attaching the reproducible test case there ?

Thanks,
Neha

On Wednesday, March 20, 2013, Íõ¹ú¶° wrote:

> Hi Jun,
>
> We use one thread with one sync produce to send data to broker
> (QPS:10k-15k, each log is about 1k bytes). The problem is reproduced.
>
> We have used Producer and SyncProducer in our test. The same Exception
> appears.
>
> Thanks.
>
>
>
> On Thu, Mar 21, 2013 at 12:19 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > How many threads are you using?
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Mar 20, 2013 at 7:33 PM, Yang Zhou <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Sorry, I made a mistake, we use many threads producing at same time.
> > >
> > >
> > > 2013/3/20 Jun Rao <[EMAIL PROTECTED]>
> > >
> > > > How many producer instances do you have? Can you reproduce the
> problem
> > > with
> > > > a single producer?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Mar 20, 2013 at 12:29 AM, Íõ¹ú¶° <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > we do not use any compression in our test.
> > > > >
> > > > > We deploy producer and broker in the same machine. The problem
> still
> > > > > exists. We use sync producer, and send one message at a time(no
> batch
> > > > now).
> > > > > We find that when the qps reaches more than 40k, the exception
> > appears.
> > > > So
> > > > > I don't think it's the underlying system error.
> > > > >
> > > > > Any suggestions if we want to do some debug on kafka
> > > > > serialization/deserialization?
> > > > >
> > > > > Thanks.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Mar 20, 2013 at 12:10 AM, Jun Rao <[EMAIL PROTECTED]>
> wrote:
> > > > >
> > > > > > It basically means that the broker is expecting to read certain
> > > number
> > > > of
> > > > > > bytes in a buffer received from socket, but there are fewer bytes
> > > than
> > > > > > expected in the buffer. Possible causes are (1) a bug in Kafka
> > > request
> > > > > > serialization/deserialization logic; (2) corruption in the
> > underlying
> > > > > > system such as network.
> > > > > >
> > > > > > BTW, did you enable compression in your producer?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Mon, Mar 18, 2013 at 10:12 PM, Helin Xiang <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > > >
> > > > > > > thanks Jun.
> > > > > > >
> > > > > > > we are using java producer.
> > > > > > > does the last exception
> > > > > > > "java.lang.IllegalArgumentException
> > > > > > >     at java.nio.Buffer.limit(Buffer.java:266)
> > > > > > > "
> > > > > > > also means the broker received corrupted messages ?  sorry i am
> > not
> > > > > > > familiar with java nio.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 19, 2013 at 12:58 PM, Jun Rao <[EMAIL PROTECTED]>
> > > wrote:
> > > > > > >
> > > > > > > > Hmm, both log4j messages suggest that the broker received
> some
> > > > > > corrupted
> > > > > > > > produce requests. Are you using the java producer? Also, we
> > have
> > > > seen
> > > > > > > that
> > > > > > > > network router problems caused corrupted requests before.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Mon, Mar 18, 2013 at 8:22 PM, Helin Xiang <
> --
> Guodong Wang
> Íõ¹ú¶°
>

 
+
王国栋 2013-03-25, 03:58
+
Neha Narkhede 2013-03-25, 04:01
+
王国栋 2013-03-25, 04:23