Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Truncated messages in fetch response


Copy link to this message
-
Re: Truncated messages in fetch response
Jay is correct. ..it manifests itself when the size threshold condition is
violated.

For erlang client,  this is fixed in 0.4.6 of mps.

Github.com/milindparikh/mps

Regards
Milind
On Jun 27, 2013 3:30 PM, "Bob Potter" <[EMAIL PROTECTED]> wrote:

> Vadim,
>
> I don't know under exactly what conditions it happens but the behavior in
> this test seems to reliably reproduce it:
>
> https://github.com/v-a/poseidon/commit/d0ac928e0967e1eaf5b92b403103c4f0dc8fd7f7
>
> -Bob
>
>
> On 27 June 2013 14:29, Vadim Keylis <[EMAIL PROTECTED]> wrote:
>
> > Jay. I assume this is problem exists in the consumer. How this can this
> > problem be triggered so I could test my high level consumer.
> >
> > Thanks
> >
> > On Jun 26, 2013, at 9:21 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> > > Yeah, that is true. I thought I documented that, but looking at the
> > > protocol docs, it looks like I didn't.
> > >
> > > I agree this is kind of a pain in the ass. It was an important
> > > optimization in 0.7 because we didn't know where the message
> > > boundaries were but in 0.8 we have a fast way to compute message
> > > boundaries and in fact we normally don't give out partial messages, I
> > > think this happens when you hit the size threshold of your fetch
> > > request (e.g. 1MB) instead of searching for the nearest message
> > > boundary we give you that chunk of log. I think we should consider
> > > just fixing it entirely in the next release--the perf hit is pretty
> > > minor and it is an annoyance and source of bugs for clients.
> > >
> > > For now you have to handle it, so I added documentation to the protocol
> > wiki.
> > >
> > > Sorry!
> > >
> > > -Jay
> > >
> > > On Wed, Jun 26, 2013 at 8:59 AM, Bob Potter <[EMAIL PROTECTED]>
> > wrote:
> > >> Howdy,
> > >>
> > >> I'm developing a client for kafka 0.8. It looks like a fetch response
> > will
> > >> sometimes end with a partial message. I understand why this might be
> the
> > >> case but it was unexpected and as far as I can tell undocumented.
> > >>
> > >> Is my understanding correct or am i missing something?
> > >>
> > >> -Bob
> >
>
>
>
> --
> Bob Potter
>