Kafka, mail # user - Kafka broker not respecting log.roll.hours? - 2013-04-25, 19:45
 Search Hadoop and all its subprojects:

Switch to Plain View
Copy link to this message
Kafka broker not respecting log.roll.hours?
We are running Kafka 0.7.2. We set log.roll.hours=1. I hoped that meant
logs would be rolled every hour, or more. Only, sometimes logs that are
many hours (sometimes days) old have more data added to them. This perturbs
our systems for reasons I won't get in to.

Have others observed this? Is it a bug? Is there a planned fix?

I don't know Scala or Kafka well, but I have proposal for why this might
happen: upon restart, a broker forgets when its log files have been
appended to ("firstAppendTime"). Then a potentially infinite amount of time
later, the restarted broker receives another message for the particular
(topic, partition), and starts the clock again. It will then roll over that
log after an hour.


  /* the maximum time before a new log segment is rolled out */
  val logRollHours = Utils.getIntInRange(props, "log.roll.hours", 24*7, (1,

maybeRoll, which needs segment.firstAppendTime defined. It also has
updateFirstAppendTime() which says if it's empty, then set it.

Jun Rao 2013-04-26, 04:49
Dan Frankowski 2013-04-26, 06:13
Jun Rao 2013-04-26, 14:40
Dan Frankowski 2013-04-26, 15:20
Jason Rosenberg 2013-04-26, 16:52
Adam Talaat 2013-04-26, 17:33
Dan Frankowski 2013-04-27, 21:37
Swapnil Ghike 2013-04-28, 09:05
Dan Frankowski 2013-05-02, 21:23
Jun Rao 2013-05-03, 15:58
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB