Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Truncated messages in fetch response


Copy link to this message
-
Re: Truncated messages in fetch response
Jay is correct. ..it manifests itself when the size threshold condition is
violated.

For erlang client,  this is fixed in 0.4.6 of mps.

Github.com/milindparikh/mps

Regards
Milind
On Jun 27, 2013 3:30 PM, "Bob Potter" <[EMAIL PROTECTED]> wrote:

> Vadim,
>
> I don't know under exactly what conditions it happens but the behavior in
> this test seems to reliably reproduce it:
>
> https://github.com/v-a/poseidon/commit/d0ac928e0967e1eaf5b92b403103c4f0dc8fd7f7
>
> -Bob
>
>
> On 27 June 2013 14:29, Vadim Keylis <[EMAIL PROTECTED]> wrote:
>
> > Jay. I assume this is problem exists in the consumer. How this can this
> > problem be triggered so I could test my high level consumer.
> >
> > Thanks
> >
> > On Jun 26, 2013, at 9:21 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> > > Yeah, that is true. I thought I documented that, but looking at the
> > > protocol docs, it looks like I didn't.
> > >
> > > I agree this is kind of a pain in the ass. It was an important
> > > optimization in 0.7 because we didn't know where the message
> > > boundaries were but in 0.8 we have a fast way to compute message
> > > boundaries and in fact we normally don't give out partial messages, I
> > > think this happens when you hit the size threshold of your fetch
> > > request (e.g. 1MB) instead of searching for the nearest message
> > > boundary we give you that chunk of log. I think we should consider
> > > just fixing it entirely in the next release--the perf hit is pretty
> > > minor and it is an annoyance and source of bugs for clients.
> > >
> > > For now you have to handle it, so I added documentation to the protocol
> > wiki.
> > >
> > > Sorry!
> > >
> > > -Jay
> > >
> > > On Wed, Jun 26, 2013 at 8:59 AM, Bob Potter <[EMAIL PROTECTED]>
> > wrote:
> > >> Howdy,
> > >>
> > >> I'm developing a client for kafka 0.8. It looks like a fetch response
> > will
> > >> sometimes end with a partial message. I understand why this might be
> the
> > >> case but it was unexpected and as far as I can tell undocumented.
> > >>
> > >> Is my understanding correct or am i missing something?
> > >>
> > >> -Bob
> >
>
>
>
> --
> Bob Potter
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB