Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka log retentions


+
Ankit Jain 2013-04-01, 19:44
+
Philip OToole 2013-04-01, 19:50
+
Jason Rosenberg 2013-04-01, 20:00
+
Yonghui Zhao 2013-04-02, 03:45
+
Philip OToole 2013-04-02, 03:50
Copy link to this message
-
Re: Kafka log retentions
Got it£¬thanks
2013/4/2 Philip O'Toole <[EMAIL PROTECTED]>

> On Mon, Apr 1, 2013 at 8:44 PM, Yonghui Zhao <[EMAIL PROTECTED]>
> wrote:
>
> > A little question£¬ when old message log is deleted£¬ the start point
> offset
> > is changed. If a simple consumer seek a offset less than start point
> > offset, what will happen? Read all message from start?
> >
>
> Standard Kafka Consumer code will raise an OffsetOutOfRange exception. How
> that is handled depends on the the type of Consumer you use -- Simple or
> High-Level. I believe the latter moves you to the next available offset.
>
> Philip
>
>
> >
> > 2013/4/2 Jason Rosenberg <[EMAIL PROTECTED]>
> >
> > > Essentially,
> > >
> > > There's a configuration property: log.retention.hours
> > >
> > > This determines the minimum time a message will remain available on the
> > > broker.  The default is 7 days.
> > >
> > > The kafka broker doesn't keep track of whether the message has been
> > > consumed or not (or how many times it has been consumed).
> > >
> > > It's up to individual consumers to keep track of which message offset
> > > they've consumed (which typically gets saved in zookeeper).
> > >
> > > Jason
> > >
> > >
> > > On Mon, Apr 1, 2013 at 12:50 PM, Philip O'Toole <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > On Mon, Apr 1, 2013 at 12:27 PM, Ankit Jain <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Once the message is consumed by consumer, we want it to delete from
> > > > message
> > > > > broker as well.
> > > > >
> > > >
> > > > Kafka doesn't work this way. Read the design doc -- it's well
> written,
> > > and
> > > > should be read by anyone working with Kafka.
> > > >
> > > > http://kafka.apache.org/design.html
> > > >
> > > >
> > > >
> > > > >
> > > > > I was exploring the kafka configuration, but not sure which
> > > configuration
> > > > > would full my need.
> > > > >
> > > > > Guys, I need your help.
> > > > >
> > > > > Thanks ..
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Ankit Jain
> > > > >
> > > >
> > >
> >
>

 
+
Jun Rao 2013-04-02, 04:50
+
Tom Brown 2013-04-02, 05:46