Is there currently a way to get the latest offset (or close to the
latest) for a partition? Is this covered by the Offset API, and if so
could someone explain this API a bit?

I'd like to write a consumer that can start from 0 (like unix cat) or
start from the most recent (like unix tail).

Thanks!
-David

 

David,

The API that you are looking for is OffsetRequest. If you pass in -2 as the
value for Time, it will get you the earliest offset available for that
partition. If you pass in -1, it will get you the latest offset.

You can find more information about the OffsetRequest and OffsetResponse
APIs here -
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI

Please let us know if that description can be improved.

Thanks,
Neha
On Wed, Feb 20, 2013 at 6:31 PM, David Arthur <[EMAIL PROTECTED]> wrote:

> Is there currently a way to get the latest offset (or close to the latest)
> for a partition? Is this covered by the Offset API, and if so could someone
> explain this API a bit?
>
> I'd like to write a consumer that can start from 0 (like unix cat) or
> start from the most recent (like unix tail).
>
> Thanks!
> -David
>

 

Yes, I think that section of the protocol guide could be improved.
Specifically, it would be useful to explain what the " Time" and
"MaxNumberOfOffsets" parameters are used for, as well documenting these
two magic numbers (-1 and -2).

-David

On 2/21/13 12:13 AM, Neha Narkhede wrote:
> David,
>
> The API that you are looking for is OffsetRequest. If you pass in -2 as the
> value for Time, it will get you the earliest offset available for that
> partition. If you pass in -1, it will get you the latest offset.
>
> You can find more information about the OffsetRequest and OffsetResponse
> APIs here -
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI
>
> Please let us know if that description can be improved.
>
> Thanks,
> Neha
>
>
> On Wed, Feb 20, 2013 at 6:31 PM, David Arthur <[EMAIL PROTECTED]> wrote:
>
>> Is there currently a way to get the latest offset (or close to the latest)
>> for a partition? Is this covered by the Offset API, and if so could someone
>> explain this API a bit?
>>
>> I'd like to write a consumer that can start from 0 (like unix cat) or
>> start from the most recent (like unix tail).
>>
>> Thanks!
>> -David
>>
 

In the old protocol description the -1 and -2 values are explicitly
mentioned.
https://cwiki.apache.org/confluence/display/KAFKA/Wire+Format

It looks like it had been left out of the new 0.8 docs, however I just
added it.
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI

Hopefully the docs are more helpful now. Feel free to ask for further
clarifications.
On 02/21/2013 05:04 AM, David Arthur wrote:
> Yes, I think that section of the protocol guide could be improved.
> Specifically, it would be useful to explain what the " Time" and
> "MaxNumberOfOffsets" parameters are used for, as well documenting these
> two magic numbers (-1 and -2).
>
> -David
>
> On 2/21/13 12:13 AM, Neha Narkhede wrote:
>> David,
>>
>> The API that you are looking for is OffsetRequest. If you pass in -2 as the
>> value for Time, it will get you the earliest offset available for that
>> partition. If you pass in -1, it will get you the latest offset.
>>
>> You can find more information about the OffsetRequest and OffsetResponse
>> APIs here -
>> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI
>>
>> Please let us know if that description can be improved.
>>
>> Thanks,
>> Neha
>>
>>
>> On Wed, Feb 20, 2013 at 6:31 PM, David Arthur <[EMAIL PROTECTED]> wrote:
>>
>>> Is there currently a way to get the latest offset (or close to the latest)
>>> for a partition? Is this covered by the Offset API, and if so could someone
>>> explain this API a bit?
>>>
>>> I'd like to write a consumer that can start from 0 (like unix cat) or
>>> start from the most recent (like unix tail).
>>>
>>> Thanks!
>>> -David
>>>
--
*Colin Blower*
/Software Engineer/
Barracuda Networks Inc.
+1 408-342-5576 (o)

 

Thank you for the feedback on documentation and also for improving it. That
is the best way to have effective documentation.

Thanks,
Neha
On Thu, Feb 21, 2013 at 8:53 AM, Colin Blower <[EMAIL PROTECTED]> wrote:

> In the old protocol description the -1 and -2 values are explicitly
> mentioned.
> https://cwiki.apache.org/**confluence/display/KAFKA/Wire+**Format<https://cwiki.apache.org/confluence/display/KAFKA/Wire+Format>
>
> It looks like it had been left out of the new 0.8 docs, however I just
> added it.
> https://cwiki.apache.org/**confluence/display/KAFKA/A+**
> Guide+To+The+Kafka+Protocol#**AGuideToTheKafkaProtocol-**OffsetAPI<https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI>
>
> Hopefully the docs are more helpful now. Feel free to ask for further
> clarifications.
>
>
>
> On 02/21/2013 05:04 AM, David Arthur wrote:
>
>> Yes, I think that section of the protocol guide could be improved.
>> Specifically, it would be useful to explain what the " Time" and
>> "MaxNumberOfOffsets" parameters are used for, as well documenting these
>> two magic numbers (-1 and -2).
>>
>> -David
>>
>> On 2/21/13 12:13 AM, Neha Narkhede wrote:
>>
>>> David,
>>>
>>> The API that you are looking for is OffsetRequest. If you pass in -2 as
>>> the
>>> value for Time, it will get you the earliest offset available for that
>>> partition. If you pass in -1, it will get you the latest offset.
>>>
>>> You can find more information about the OffsetRequest and OffsetResponse
>>> APIs here -
>>> https://cwiki.apache.org/**confluence/display/KAFKA/A+**
>>> Guide+To+The+Kafka+Protocol#**AGuideToTheKafkaProtocol-**OffsetAPI<https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI>
>>>
>>> Please let us know if that description can be improved.
>>>
>>> Thanks,
>>> Neha
>>>
>>>
>>> On Wed, Feb 20, 2013 at 6:31 PM, David Arthur <[EMAIL PROTECTED]> wrote:
>>>
>>>  Is there currently a way to get the latest offset (or close to the
>>>> latest)
>>>> for a partition? Is this covered by the Offset API, and if so could
>>>> someone
>>>> explain this API a bit?
>>>>
>>>> I'd like to write a consumer that can start from 0 (like unix cat) or
>>>> start from the most recent (like unix tail).
>>>>
>>>> Thanks!
>>>> -David
>>>>
>>>>
>
> --
> *Colin Blower*
> /Software Engineer/
> Barracuda Networks Inc.
> +1 408-342-5576 (o)
>