Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Consume from X messages ago


+
James Englert 2013-03-19, 15:05
+
David Arthur 2013-03-19, 15:17
+
James Englert 2013-03-19, 15:26
Copy link to this message
-
Re: Consume from X messages ago
David Arthur 2013-03-19, 15:49
This API is exposed through the SimpleConsumer scala class. See
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/consumer/SimpleConsumer.scala#L60

You will need to set earliestOrLatest to -1 for the latest offset.

There is also a command line tool
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/GetOffsetShell.scala

-David

On 3/19/13 11:25 AM, James Englert wrote:
> I'm still a bit lost.  Where is the offsets API?  I.e. which class?
>
>
> On Tue, Mar 19, 2013 at 11:16 AM, David Arthur <[EMAIL PROTECTED]> wrote:
>
>> Using the Offsets API, you can get the latest offset by setting time to
>> -1. Then you subtract 10000
>>
>> There is no guarantee that 10k prior messages exist of course, so you'd
>> need to handle that case.
>>
>> -David
>>
>>
>> On 3/19/13 11:04 AM, James Englert wrote:
>>
>>> Hi,
>>>
>>> I'm using Kafka 0.8.  I would like to setup a consumer to fetch the last
>>> 10,000 messages and then start consuming messages.
>>>
>>> I see the configuration autooffset.reset, but that isn't quite what I
>>> want.  I want only the last 10,000 messages.
>>>
>>> Is there a good way to achieve this in 0.8, besides just hacking the data
>>> in ZK?
>>>
>>> Thanks,
>>> Jim
>>>
>>>
>
 
+
Neha Narkhede 2013-03-19, 16:16
+
S Ahmed 2013-03-19, 18:01
+
Neha Narkhede 2013-03-19, 18:05
+
James Englert 2013-03-22, 15:13