Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> zero size ".index" files


Copy link to this message
-
Re: zero size ".index" files
Take a look at log.index.size.max.bytes described in
http://kafka.apache.org/documentation.html#brokerconfigs

Thanks,

Jun
On Sun, Nov 3, 2013 at 4:55 PM, lee yeonok <[EMAIL PROTECTED]> wrote:

> Hi, all.
> I am not quite understood how *.index files are used or created.
>
> I started a broker1, and set the property "log.segment.bytes" to "100".
> Here's a file tree in log directory what I've got after that broker1
> received some messages.
>
> kafka-logs-1/
> |-- preview-0
> |   |-- 00000000000000000000.index
> |   |-- 00000000000000000000.log
> |   |-- 00000000000000000031.index
> |   |-- 00000000000000000031.log
> |   |-- 00000000000000000033.index
> |   |-- 00000000000000000033.log
> |   |-- 00000000000000000035.index
> |   |-- 00000000000000000035.log
> |   |-- 00000000000000000037.index
> |   |-- 00000000000000000037.log
> |   |-- 00000000000000000039.index
> |   `-- 00000000000000000039.log
>
>
> At the point the segment size reached at 100 bytes, a new file is created
> as I expected, however, I am not sure why the old *.index files are being
> truncated.
> In my log directory,  00000000000000000000.index,
>  00000000000000000031.index, 00000000000000000033.index,
> 00000000000000000035.index,
> 00000000000000000037.index are zero size file. Is that all right?
>
> I thought that in this case the consumers are not able to get the messages
> from beginning since the old index files are being truncated, however I
> suspect that there's no problem to consume the messages from beginning.
>
> I removed all the index files, and found that when the broker get new
> message the latest index file is recovered. At this moment the old index
> files doesn't exists though, consuming all messages from beginning is not
> problem.
>
> So my conclusion is this Kafka only has the latest index, and in the case
> the old message requested Kafka re-parse every log files(not refer index
> file).
> Is it right? If then, having a zero size index files is unnecessary....
> Why does Kafka recreate all the missing index files when the broker shut
> down?
> I shut down a broker1, and the removed index files are recreated.
>
>
> --
> yeonok lee
>