lee yeonok 2013-11-04, 00:56
Take a look at log.index.size.max.bytes described in
On Sun, Nov 3, 2013 at 4:55 PM, lee yeonok <[EMAIL PROTECTED]> wrote:
> Hi, all.
> I am not quite understood how *.index files are used or created.
> I started a broker1, and set the property "log.segment.bytes" to "100".
> Here's a file tree in log directory what I've got after that broker1
> received some messages.
> |-- preview-0
> | |-- 00000000000000000000.index
> | |-- 00000000000000000000.log
> | |-- 00000000000000000031.index
> | |-- 00000000000000000031.log
> | |-- 00000000000000000033.index
> | |-- 00000000000000000033.log
> | |-- 00000000000000000035.index
> | |-- 00000000000000000035.log
> | |-- 00000000000000000037.index
> | |-- 00000000000000000037.log
> | |-- 00000000000000000039.index
> | `-- 00000000000000000039.log
> At the point the segment size reached at 100 bytes, a new file is created
> as I expected, however, I am not sure why the old *.index files are being
> In my log directory, 00000000000000000000.index,
> 00000000000000000031.index, 00000000000000000033.index,
> 00000000000000000037.index are zero size file. Is that all right?
> I thought that in this case the consumers are not able to get the messages
> from beginning since the old index files are being truncated, however I
> suspect that there's no problem to consume the messages from beginning.
> I removed all the index files, and found that when the broker get new
> message the latest index file is recovered. At this moment the old index
> files doesn't exists though, consuming all messages from beginning is not
> So my conclusion is this Kafka only has the latest index, and in the case
> the old message requested Kafka re-parse every log files(not refer index
> Is it right? If then, having a zero size index files is unnecessary....
> Why does Kafka recreate all the missing index files when the broker shut
> I shut down a broker1, and the removed index files are recreated.
> yeonok lee