Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka broker not respecting log.roll.hours?


Copy link to this message
-
Kafka broker not respecting log.roll.hours?
We are running Kafka 0.7.2. We set log.roll.hours=1. I hoped that meant
logs would be rolled every hour, or more. Only, sometimes logs that are
many hours (sometimes days) old have more data added to them. This perturbs
our systems for reasons I won't get in to.

Have others observed this? Is it a bug? Is there a planned fix?

I don't know Scala or Kafka well, but I have proposal for why this might
happen: upon restart, a broker forgets when its log files have been
appended to ("firstAppendTime"). Then a potentially infinite amount of time
later, the restarted broker receives another message for the particular
(topic, partition), and starts the clock again. It will then roll over that
log after an hour.

https://svn.apache.org/repos/asf/kafka/branches/0.7/core/src/main/scala/kafka/server/KafkaConfig.scalasays:

  /* the maximum time before a new log segment is rolled out */
  val logRollHours = Utils.getIntInRange(props, "log.roll.hours", 24*7, (1,
Int.MaxValue))

https://svn.apache.org/repos/asf/kafka/branches/0.7/core/src/main/scala/kafka/log/Log.scalahas
maybeRoll, which needs segment.firstAppendTime defined. It also has
updateFirstAppendTime() which says if it's empty, then set it.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB