Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka throw InvalidMessageException and lost data


+
Helin Xiang 2013-03-19, 03:22
+
Jun Rao 2013-03-19, 04:58
+
Helin Xiang 2013-03-19, 05:13
+
Neha Narkhede 2013-03-19, 05:55
+
王国栋 2013-03-20, 07:34
+
Jun Rao 2013-03-19, 16:10
+
王国栋 2013-03-20, 07:29
+
Jun Rao 2013-03-20, 14:29
+
Yang Zhou 2013-03-21, 02:09
+
Yang Zhou 2013-03-21, 02:33
+
Jun Rao 2013-03-21, 04:20
+
王国栋 2013-03-21, 04:43
+
Neha Narkhede 2013-03-21, 05:05
+
王国栋 2013-03-25, 03:58
Copy link to this message
-
Re: Kafka throw InvalidMessageException and lost data
That's great. Would you know the details of the bug though ? It will be
helpful for the community to understand.

Thanks,
Neha

On Sunday, March 24, 2013, 王国栋 wrote:

> Hi Neha & Jun,
>
> I think we have found the reason of this bug.
> It is related to jdk versions. In the beginning, we use jdk1.6.20 to run
> the test.
>
> Yesterday, we used jdk1.6.38. Everything is OK now.
>
> Thanks a lot for all your kind help and answer.
>
>
>
> On Thu, Mar 21, 2013 at 1:05 PM, Neha Narkhede <[EMAIL PROTECTED]<javascript:;>
> >wrote:
>
> > Do you mind filing a bug and attaching the reproducible test case there ?
> >
> > Thanks,
> > Neha
> >
> > On Wednesday, March 20, 2013, 王国栋 wrote:
> >
> > > Hi Jun,
> > >
> > > We use one thread with one sync produce to send data to broker
> > > (QPS:10k-15k, each log is about 1k bytes). The problem is reproduced.
> > >
> > > We have used Producer and SyncProducer in our test. The same Exception
> > > appears.
> > >
> > > Thanks.
> > >
> > >
> > >
> > > On Thu, Mar 21, 2013 at 12:19 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > How many threads are you using?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Mar 20, 2013 at 7:33 PM, Yang Zhou <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > >
> > > > > Sorry, I made a mistake, we use many threads producing at same
> time.
> > > > >
> > > > >
> > > > > 2013/3/20 Jun Rao <[EMAIL PROTECTED]>
> > > > >
> > > > > > How many producer instances do you have? Can you reproduce the
> > > problem
> > > > > with
> > > > > > a single producer?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Mar 20, 2013 at 12:29 AM, 王国栋 <[EMAIL PROTECTED]>
> wrote:
> > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > we do not use any compression in our test.
> > > > > > >
> > > > > > > We deploy producer and broker in the same machine. The problem
> > > still
> > > > > > > exists. We use sync producer, and send one message at a time(no
> > > batch
> > > > > > now).
> > > > > > > We find that when the qps reaches more than 40k, the exception
> > > > appears.
> > > > > > So
> > > > > > > I don't think it's the underlying system error.
> > > > > > >
> > > > > > > Any suggestions if we want to do some debug on kafka
> > > > > > > serialization/deserialization?
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Mar 20, 2013 at 12:10 AM, Jun Rao <[EMAIL PROTECTED]>
> > > wrote:
> > > > > > >
> > > > > > > > It basically means that the broker is expecting to read
> certain
> > > > > number
> > > > > > of
> > > > > > > > bytes in a buffer received from socket, but there are fewer
> > bytes
> > > > > than
> > > > > > > > expected in the buffer. Possible causes are (1) a bug in
> Kafka
> > > > > request
> > > > > > > > serialization/deserialization logic; (2) corruption in the
> > > > underlying
> > > > > > > > system such as network.
> > > > > > > >
> > > > > > > > BTW, did you enable compression in your producer?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Mon, Mar 18, 2013 at 10:12 PM, Helin Xiang <
> > [EMAIL PROTECTED]
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > thanks Jun.
> > > > > > > > >
> > > > > > > > > we are using java producer.
> > > > > > > > > does the last exception
> > > > > > > > > "java.lang.IllegalArgumentException<--
> Guodong Wang
> 王国栋
>

 
+
王国栋 2013-03-25, 04:23
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB