Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - kafka and ruby client, beginner questions


Copy link to this message
-
Re: kafka and ruby client, beginner questions
Felix GV 2012-05-08, 18:24
The way I understand it, if you batch your messages (by default, the
setting is still 1, I think, so no batching) and you have compression
enabled, then each valid offset does correspond to a whole batch of
messages, which is what you were referring to, I think.

Keep in mind that I have not played around with compression yet, so take
whatever I say with a grain of salt.

Also, if you have batching (>1) enabled, but no compression, I would assume
that you can still get each individual message in a batch with its
appropriate offset, rather than only being able to get an offset for the
entire batch of messages, but I have not tested that.

Sorry for not being able to provide any definitive answer. Hopefully, an
expert can chip in to confirm or infirm what I said...!

--
Felix

On Mon, May 7, 2012 at 9:57 AM, S Ahmed <[EMAIL PROTECTED]> wrote:

> Ok sorry I think I was confused.
>
> Once I have the offset, say I have the first offset (would it be 0?), I
> then call consume and it gets me the next message, and I keep calling
> consume till it reaches the end.
>
>
> On Mon, May 7, 2012 at 9:52 AM, Sebastian Eichner <
> [EMAIL PROTECTED]
> > wrote:
>
> > On Mon, May 7, 2012 at 3:47 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> > > Is it possible to query for a list of offsets?
> >
> > Your Ruby Consumer has exactly 1 offset. You can query it via
> > consumer.offset (maybe take a look at the gem's sourcecode, its easy
> > to understand).
> >
> > > From what I understand, each offset is not for a single message, but
> > could
> > > be a group of messages right?
> >
> > Not sure what you are referring to, but maybe you mix up the info on
> > the Java Consumer, which handles multiple streams in parallel (and
> > then does have multiple offsets). The Ruby consumer is much simpler
> > (like the Java SimpleConsumer) and just works single-threaded with one
> > offset.
> >
> > Sebastian
> >
>