Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Determining offset of last produce?


Copy link to this message
-
Re: Determining offset of last produce?
Yes - take a look at the GetOffsetShell tool. It may also help to look at
the SimpleConsumer class' getOffsetsBefore method.

Thanks,

Joel

On Wed, Aug 8, 2012 at 12:42 PM, Brian O'Neill <[EMAIL PROTECTED]> wrote:

>
> Thanks for the response Joel.  That will work for now.
>
> I'm queuing a batch of data. (for a Storm topology)
> I really just need to know the consuming offset that I need to wait for to
> know that the batch is complete.  As long as I get a offset greater than
> that of the last message I queued, I should be good.  (So I should be good
> even w/ multiple producers)
>
> Is there an example somewhere for issuing an OffSetRequest?  I grep'd the
> codebase, but only see references in the Hadoop contribution.
>
> -brian
>
>
> --
> Brian O'Neill
> Lead Architect, Software Development
> Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
> p: 215.588.6024
> www.healthmarketscience.com
>
>
>
>
>
> On 8/8/12 2:15 PM, "Joel Koshy" <[EMAIL PROTECTED]> wrote:
>
> >Hi Brian,
> >
> >This is not directly available in 0.7 which does not have any producer
> >acks. The closest I can think of right now is to issue offset requests
> >after your producer send, but that would make sense if you only have one
> >producer to that topic
> >
> >0.8 which is under development has producer acks which does return the
> >next
> >append offsets.
> >
> >Joel
> >
> >On Wed, Aug 8, 2012 at 10:29 AM, Brian O'Neill
> ><[EMAIL PROTECTED]>wrote:
> >
> >> noob question:
> >> What's the best way to determine the offset of the message I just
> >> produced via the java api?
> >>
> >> -brian
> >>
> >> --
> >> Brian ONeill
> >> Lead Architect, Health Market Science (http://healthmarketscience.com)
> >> mobile:215.588.6024
> >> blog: http://weblogs.java.net/blog/boneill42/
> >> blog: http://brianoneill.blogspot.com/
> >>
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB