Hi,

We are sending all of our production events to Kafka, but nobody is
consuming it right now. The system is running for a week.
The retention period is set to 72 hours.

When I run a SimpleConsumer with offset 0, I get OffsetOutOfRangeException.
I tried using some arbitrary higher values such as 1000000 etc, but it
didn't work.
It was working fine few days ago. If I use Zookeeper based consumer, it
works fine.

Is there any way I can find out what is the least offset for a topic on the
broker and start with that using  SimpleConsumer? If I don't know the exact
offset, am I not allowed to use SimpleConsumer?
Is this happening because some of the events might have been deleted by
borkers given that the log retention period is only 72 hours?

Regards,
Vaibhav
GumGum

> The retention period is set to 72 hours.
>
> When I run a SimpleConsumer with offset 0, I get OffsetOutOfRangeException.
>

That offset is probably no longer available.
> I tried using some arbitrary higher values such as 1000000 etc, but it
> didn't work.
>

The offsets need to be at valid message boundaries.
> It was working fine few days ago. If I use Zookeeper based consumer, it
> works fine.
>
> Is there any way I can find out what is the least offset for a topic on the
> broker and start with that using  SimpleConsumer? If I don't know the exact
> offset, am I not allowed to use SimpleConsumer?
>
>
You can use SimpleConsumer if you wish although it is more difficult to use
(since you have to deal with offsets) than the Zookeeper based consumer.

Anyway, to find available offsets you can use the GetOffsetShell:

kafka-run-class.sh kafka.tools.GetOffsetShell --server kafka://broker:port
--topic topic --partition 0 --time -2
-2 means earliest offset available on this broker
-1 means latest offset available on this broker

Thanks,

Joel

Thanks Joel.

I actually dont want to deal with offsets. The Kafka storm spout is forcing
me to deal with it :)

Regards,
Vaibhav,
GumGum
On Jul 3, 2012 6:51 PM, "Joel Koshy" <[EMAIL PROTECTED]> wrote:

> > The retention period is set to 72 hours.
> >
> > When I run a SimpleConsumer with offset 0, I get
> OffsetOutOfRangeException.
> >
>
> That offset is probably no longer available.
>
>
> > I tried using some arbitrary higher values such as 1000000 etc, but it
> > didn't work.
> >
>
> The offsets need to be at valid message boundaries.
>
>
> > It was working fine few days ago. If I use Zookeeper based consumer, it
> > works fine.
> >
> > Is there any way I can find out what is the least offset for a topic on
> the
> > broker and start with that using  SimpleConsumer? If I don't know the
> exact
> > offset, am I not allowed to use SimpleConsumer?
> >
> >
> You can use SimpleConsumer if you wish although it is more difficult to use
> (since you have to deal with offsets) than the Zookeeper based consumer.
>
> Anyway, to find available offsets you can use the GetOffsetShell:
>
> kafka-run-class.sh kafka.tools.GetOffsetShell --server kafka://broker:port
> --topic topic --partition 0 --time -2
> -2 means earliest offset available on this broker
> -1 means latest offset available on this broker
>
> Thanks,
>
> Joel
>