Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka log retentions


Copy link to this message
-
Re: Kafka log retentions
If you use the high-level consumer, when the consumer gets an
OffsetOutofRangeException, it will automatically reset to either the
smallest or the largest available offset, depend on the config
autooffset.reset.

Thanks,

Jun

On Mon, Apr 1, 2013 at 8:44 PM, Yonghui Zhao <[EMAIL PROTECTED]> wrote:

> A little question, when old message log is deleted, the start point offset
> is changed. If a simple consumer seek a offset less than start point
> offset, what will happen? Read all message from start?
>
>
> 2013/4/2 Jason Rosenberg <[EMAIL PROTECTED]>
>
> > Essentially,
> >
> > There's a configuration property: log.retention.hours
> >
> > This determines the minimum time a message will remain available on the
> > broker.  The default is 7 days.
> >
> > The kafka broker doesn't keep track of whether the message has been
> > consumed or not (or how many times it has been consumed).
> >
> > It's up to individual consumers to keep track of which message offset
> > they've consumed (which typically gets saved in zookeeper).
> >
> > Jason
> >
> >
> > On Mon, Apr 1, 2013 at 12:50 PM, Philip O'Toole <[EMAIL PROTECTED]>
> wrote:
> >
> > > On Mon, Apr 1, 2013 at 12:27 PM, Ankit Jain <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Once the message is consumed by consumer, we want it to delete from
> > > message
> > > > broker as well.
> > > >
> > >
> > > Kafka doesn't work this way. Read the design doc -- it's well written,
> > and
> > > should be read by anyone working with Kafka.
> > >
> > > http://kafka.apache.org/design.html
> > >
> > >
> > >
> > > >
> > > > I was exploring the kafka configuration, but not sure which
> > configuration
> > > > would full my need.
> > > >
> > > > Guys, I need your help.
> > > >
> > > > Thanks ..
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Ankit Jain
> > > >
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB