-Re: kafka and ruby client, beginner questions
Felix GV 2012-05-08, 18:24
The way I understand it, if you batch your messages (by default, the
setting is still 1, I think, so no batching) and you have compression
enabled, then each valid offset does correspond to a whole batch of
messages, which is what you were referring to, I think.
Keep in mind that I have not played around with compression yet, so take
whatever I say with a grain of salt.
Also, if you have batching (>1) enabled, but no compression, I would assume
that you can still get each individual message in a batch with its
appropriate offset, rather than only being able to get an offset for the
entire batch of messages, but I have not tested that.
Sorry for not being able to provide any definitive answer. Hopefully, an
expert can chip in to confirm or infirm what I said...!
On Mon, May 7, 2012 at 9:57 AM, S Ahmed <[EMAIL PROTECTED]> wrote:
> Ok sorry I think I was confused.
> Once I have the offset, say I have the first offset (would it be 0?), I
> then call consume and it gets me the next message, and I keep calling
> consume till it reaches the end.
> On Mon, May 7, 2012 at 9:52 AM, Sebastian Eichner <
> [EMAIL PROTECTED]
> > wrote:
> > On Mon, May 7, 2012 at 3:47 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> > > Is it possible to query for a list of offsets?
> > Your Ruby Consumer has exactly 1 offset. You can query it via
> > consumer.offset (maybe take a look at the gem's sourcecode, its easy
> > to understand).
> > > From what I understand, each offset is not for a single message, but
> > could
> > > be a group of messages right?
> > Not sure what you are referring to, but maybe you mix up the info on
> > the Java Consumer, which handles multiple streams in parallel (and
> > then does have multiple offsets). The Ruby consumer is much simpler
> > (like the Java SimpleConsumer) and just works single-threaded with one
> > offset.
> > Sebastian