Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka broker not respecting log.roll.hours?


+
Dan Frankowski 2013-04-25, 19:45
Copy link to this message
-
Re: Kafka broker not respecting log.roll.hours?
That logic in 0.7.2 seems correct. Basically, firstAppendTime is set on
first append to a log segment. Then, later on, when a new message is
appended and the elapsed time since firstAppendTime is larger than the roll
time, a new segment is rolled. Is your data constantly being produced?

Thanks,

Jun
On Thu, Apr 25, 2013 at 12:44 PM, Dan Frankowski <[EMAIL PROTECTED]> wrote:

> We are running Kafka 0.7.2. We set log.roll.hours=1. I hoped that meant
> logs would be rolled every hour, or more. Only, sometimes logs that are
> many hours (sometimes days) old have more data added to them. This perturbs
> our systems for reasons I won't get in to.
>
> Have others observed this? Is it a bug? Is there a planned fix?
>
> I don't know Scala or Kafka well, but I have proposal for why this might
> happen: upon restart, a broker forgets when its log files have been
> appended to ("firstAppendTime"). Then a potentially infinite amount of time
> later, the restarted broker receives another message for the particular
> (topic, partition), and starts the clock again. It will then roll over that
> log after an hour.
>
>
> https://svn.apache.org/repos/asf/kafka/branches/0.7/core/src/main/scala/kafka/server/KafkaConfig.scalasays
> :
>
>   /* the maximum time before a new log segment is rolled out */
>   val logRollHours = Utils.getIntInRange(props, "log.roll.hours", 24*7, (1,
> Int.MaxValue))
>
>
> https://svn.apache.org/repos/asf/kafka/branches/0.7/core/src/main/scala/kafka/log/Log.scalahas
> maybeRoll, which needs segment.firstAppendTime defined. It also has
> updateFirstAppendTime() which says if it's empty, then set it.
>

 
+
Dan Frankowski 2013-04-26, 06:13
+
Jun Rao 2013-04-26, 14:40
+
Dan Frankowski 2013-04-26, 15:20
+
Jason Rosenberg 2013-04-26, 16:52
+
Adam Talaat 2013-04-26, 17:33
+
Dan Frankowski 2013-04-27, 21:37
+
Dan Frankowski 2013-05-02, 21:23
+
Jun Rao 2013-05-03, 15:58
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB