Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> zero size ".index" files


+
lee yeonok 2013-11-04, 00:56
Copy link to this message
-
Re: zero size ".index" files
Take a look at log.index.size.max.bytes described in
http://kafka.apache.org/documentation.html#brokerconfigs

Thanks,

Jun
On Sun, Nov 3, 2013 at 4:55 PM, lee yeonok <[EMAIL PROTECTED]> wrote:

> Hi, all.
> I am not quite understood how *.index files are used or created.
>
> I started a broker1, and set the property "log.segment.bytes" to "100".
> Here's a file tree in log directory what I've got after that broker1
> received some messages.
>
> kafka-logs-1/
> |-- preview-0
> |   |-- 00000000000000000000.index
> |   |-- 00000000000000000000.log
> |   |-- 00000000000000000031.index
> |   |-- 00000000000000000031.log
> |   |-- 00000000000000000033.index
> |   |-- 00000000000000000033.log
> |   |-- 00000000000000000035.index
> |   |-- 00000000000000000035.log
> |   |-- 00000000000000000037.index
> |   |-- 00000000000000000037.log
> |   |-- 00000000000000000039.index
> |   `-- 00000000000000000039.log
>
>
> At the point the segment size reached at 100 bytes, a new file is created
> as I expected, however, I am not sure why the old *.index files are being
> truncated.
> In my log directory,  00000000000000000000.index,
>  00000000000000000031.index, 00000000000000000033.index,
> 00000000000000000035.index,
> 00000000000000000037.index are zero size file. Is that all right?
>
> I thought that in this case the consumers are not able to get the messages
> from beginning since the old index files are being truncated, however I
> suspect that there's no problem to consume the messages from beginning.
>
> I removed all the index files, and found that when the broker get new
> message the latest index file is recovered. At this moment the old index
> files doesn't exists though, consuming all messages from beginning is not
> problem.
>
> So my conclusion is this Kafka only has the latest index, and in the case
> the old message requested Kafka re-parse every log files(not refer index
> file).
> Is it right? If then, having a zero size index files is unnecessary....
> Why does Kafka recreate all the missing index files when the broker shut
> down?
> I shut down a broker1, and the removed index files are recreated.
>
>
> --
> yeonok lee
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB