Kafka, mail # user - Re: Kafka broker not respecting log.roll.hours? - 2013-04-26, 06:13
 Search Hadoop and all its subprojects:

Switch to Plain View
Dan Frankowski 2013-04-25, 19:45
Jun Rao 2013-04-26, 04:49
Copy link to this message
Re: Kafka broker not respecting log.roll.hours?
We have high-volume topics and low-volume topics. The problem occurs more
often for low-volume topics to be sure.

But if my hypothesis is correct about why it is happening, here is a case
where rolling is longer than an hour, even on a high volume topic:

- write to a topic for 20 minutes
- restart the broker
- wait for 5 days
- write to a topic for 20 minutes
- restart the broker
- write to a topic for an hour

The rollover time was now 5 days, 1 hour, 40 minutes. You can make it as
long as you want. Does this make sense?

We would like the rollover time to be no more than an hour, even if the
broker is restarted, or the topic is low-volume.

The cleanest way to do that might be to roll over on the hour no matter
when the file was started. That would be too fast sometimes, but that's
fine. A second way would be to embed the first append time in the file
name. A third way (not perfect, but an approximation at least) would be to
not to write to a segment if firstAppendTime is not defined and the
timestamp on the file is more than an hour old. There are probably other

What say you?
On Thu, Apr 25, 2013 at 9:49 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
Jun Rao 2013-04-26, 14:40
Dan Frankowski 2013-04-26, 15:20
Jason Rosenberg 2013-04-26, 16:52
Adam Talaat 2013-04-26, 17:33
Dan Frankowski 2013-04-27, 21:37
Swapnil Ghike 2013-04-28, 09:05
Dan Frankowski 2013-05-02, 21:23
Jun Rao 2013-05-03, 15:58
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB