Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Determining offset of last produce?


Copy link to this message
-
Re: Determining offset of last produce?
Joel Koshy 2012-08-08, 19:59
Yes - take a look at the GetOffsetShell tool. It may also help to look at
the SimpleConsumer class' getOffsetsBefore method.

Thanks,

Joel

On Wed, Aug 8, 2012 at 12:42 PM, Brian O'Neill <[EMAIL PROTECTED]> wrote:

>
> Thanks for the response Joel.  That will work for now.
>
> I'm queuing a batch of data. (for a Storm topology)
> I really just need to know the consuming offset that I need to wait for to
> know that the batch is complete.  As long as I get a offset greater than
> that of the last message I queued, I should be good.  (So I should be good
> even w/ multiple producers)
>
> Is there an example somewhere for issuing an OffSetRequest?  I grep'd the
> codebase, but only see references in the Hadoop contribution.
>
> -brian
>
>
> --
> Brian O'Neill
> Lead Architect, Software Development
> Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
> p: 215.588.6024
> www.healthmarketscience.com
>
>
>
>
>
> On 8/8/12 2:15 PM, "Joel Koshy" <[EMAIL PROTECTED]> wrote:
>
> >Hi Brian,
> >
> >This is not directly available in 0.7 which does not have any producer
> >acks. The closest I can think of right now is to issue offset requests
> >after your producer send, but that would make sense if you only have one
> >producer to that topic
> >
> >0.8 which is under development has producer acks which does return the
> >next
> >append offsets.
> >
> >Joel
> >
> >On Wed, Aug 8, 2012 at 10:29 AM, Brian O'Neill
> ><[EMAIL PROTECTED]>wrote:
> >
> >> noob question:
> >> What's the best way to determine the offset of the message I just
> >> produced via the java api?
> >>
> >> -brian
> >>
> >> --
> >> Brian ONeill
> >> Lead Architect, Health Market Science (http://healthmarketscience.com)
> >> mobile:215.588.6024
> >> blog: http://weblogs.java.net/blog/boneill42/
> >> blog: http://brianoneill.blogspot.com/
> >>
>
>
>