Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> kafka and ruby client, beginner questions


Copy link to this message
-
Re: kafka and ruby client, beginner questions
The way I understand it, if you batch your messages (by default, the
setting is still 1, I think, so no batching) and you have compression
enabled, then each valid offset does correspond to a whole batch of
messages, which is what you were referring to, I think.

Keep in mind that I have not played around with compression yet, so take
whatever I say with a grain of salt.

Also, if you have batching (>1) enabled, but no compression, I would assume
that you can still get each individual message in a batch with its
appropriate offset, rather than only being able to get an offset for the
entire batch of messages, but I have not tested that.

Sorry for not being able to provide any definitive answer. Hopefully, an
expert can chip in to confirm or infirm what I said...!

--
Felix

On Mon, May 7, 2012 at 9:57 AM, S Ahmed <[EMAIL PROTECTED]> wrote:

> Ok sorry I think I was confused.
>
> Once I have the offset, say I have the first offset (would it be 0?), I
> then call consume and it gets me the next message, and I keep calling
> consume till it reaches the end.
>
>
> On Mon, May 7, 2012 at 9:52 AM, Sebastian Eichner <
> [EMAIL PROTECTED]
> > wrote:
>
> > On Mon, May 7, 2012 at 3:47 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> > > Is it possible to query for a list of offsets?
> >
> > Your Ruby Consumer has exactly 1 offset. You can query it via
> > consumer.offset (maybe take a look at the gem's sourcecode, its easy
> > to understand).
> >
> > > From what I understand, each offset is not for a single message, but
> > could
> > > be a group of messages right?
> >
> > Not sure what you are referring to, but maybe you mix up the info on
> > the Java Consumer, which handles multiple streams in parallel (and
> > then does have multiple offsets). The Ruby consumer is much simpler
> > (like the Java SimpleConsumer) and just works single-threaded with one
> > offset.
> >
> > Sebastian
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB