Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> OffsetOutOfRangeException and getOffsetsBefore


Copy link to this message
-
Re: OffsetOutOfRangeException and getOffsetsBefore
If you use the Java API you can do SimpleConsumer.getOffsetsBefore(topic, partition, time, 1) which will return a long offset value

The time parameter can be kafka.api.OffsetRequest.EarliestTime() or kafka.api.OffsetRequest.LatestTime() based on your application need.
On May 24, 2013, at 2:03 PM, Sining Ma <[EMAIL PROTECTED]> wrote:

> Thanks Suyog
> Could you explain more about OffsetRequest?
> I find new
> OffsetRequest(topic: String, partition: Int, time: Long, maxNumOffsets: Int)
> in Kafka api.
>
>
> How can I send this request? And where can I receive a response from this OffsetRequest?
> Could you give me an example for this API?
>
>
>
> -- Regards
>   Sining Ma
>
>
>
>
>
> -----Original Message-----
> From: Suyog Rao <[EMAIL PROTECTED]>
> To: users <[EMAIL PROTECTED]>
> Sent: Fri, May 24, 2013 1:32 pm
> Subject: Re: OffsetOutOfRangeException and getOffsetsBefore
>
>
> Since you are using the simple consumer you will need to handle the
> OffsetOutOfRange Exception in your code. This happens when your consumer queries
> for an offset which is no longer persisted in Kafka (The logs have been deleted
> based on the retention policy). Ideally when this happens, the consumer should
> issue an OffsetRequest to get the latest/earliest offset currently available on
> the server. You can then use this in your FetchRequest. Whether you use the
> latest or earliest depends on your application needs. In essence, you are
> resetting the consumer to catch up with the server.
>
> On May 24, 2013, at 11:29 AM, Sining Ma <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>> We are currently using kafka-0.7.1 right now.
>> I have two questions:
>> 1. We use SimpleConsumer to aggregate messages to log files and there is no
> zookeeper. Sometimes we can see kafka.common.OffsetOutOfRangeException.
>> And this exception happens when we start our consumer program. We do not know
> the reason why this happens.
>> How can I get a valid latest message offset in kafka-0.7.1 when this exception
> happens?
>> 2. Before we start consumer, we call getOffsetsBefore function to get a list
> of valid offsets (up to maxSize) before the given time.
>> How can we interpret this list?
>> For example, this function returns an array [offset1, offset2].
>> Does this mean from offset1 to offset2 are valid, and offset2 to current
> offset are valid? We are confused about the meaning of this array.
>>
>>
>>
>> -- Regards
>>  Sining Ma
>>
>>
>
>
>