Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka log retentions


Copy link to this message
-
Re: Kafka log retentions
Philip O'Toole 2013-04-02, 03:50
On Mon, Apr 1, 2013 at 8:44 PM, Yonghui Zhao <[EMAIL PROTECTED]> wrote:

> A little question, when old message log is deleted, the start point offset
> is changed. If a simple consumer seek a offset less than start point
> offset, what will happen? Read all message from start?
>

Standard Kafka Consumer code will raise an OffsetOutOfRange exception. How
that is handled depends on the the type of Consumer you use -- Simple or
High-Level. I believe the latter moves you to the next available offset.

Philip
>
> 2013/4/2 Jason Rosenberg <[EMAIL PROTECTED]>
>
> > Essentially,
> >
> > There's a configuration property: log.retention.hours
> >
> > This determines the minimum time a message will remain available on the
> > broker.  The default is 7 days.
> >
> > The kafka broker doesn't keep track of whether the message has been
> > consumed or not (or how many times it has been consumed).
> >
> > It's up to individual consumers to keep track of which message offset
> > they've consumed (which typically gets saved in zookeeper).
> >
> > Jason
> >
> >
> > On Mon, Apr 1, 2013 at 12:50 PM, Philip O'Toole <[EMAIL PROTECTED]>
> wrote:
> >
> > > On Mon, Apr 1, 2013 at 12:27 PM, Ankit Jain <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Once the message is consumed by consumer, we want it to delete from
> > > message
> > > > broker as well.
> > > >
> > >
> > > Kafka doesn't work this way. Read the design doc -- it's well written,
> > and
> > > should be read by anyone working with Kafka.
> > >
> > > http://kafka.apache.org/design.html
> > >
> > >
> > >
> > > >
> > > > I was exploring the kafka configuration, but not sure which
> > configuration
> > > > would full my need.
> > > >
> > > > Guys, I need your help.
> > > >
> > > > Thanks ..
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Ankit Jain
> > > >
> > >
> >
>