Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Keeping logs forever


Copy link to this message
-
Keeping logs forever
Our use case is that we'd like to log data we don't need away and
potentially replay it at some point. We don't want to delete old logs. I
googled around a bit and I only discovered this particular post:
http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201210.mbox/%[EMAIL PROTECTED]%3E

In summary, it appears the primary issue is that Kafka keeps file
handles of each log segment open. Is there a way to configure this, or
is a way to do so planned? It appears that an option to deduplicate
instead of delete was added recently, so doesn't the file handle issue
exist with that as well (since files aren't being deleted)?

 
+
Eric Tschetter 2013-02-22, 00:30
+
Milind Parikh 2013-02-22, 00:44
+
Jay Kreps 2013-02-22, 01:26
+
graham sanderson 2013-02-22, 02:47
+
Eric Tschetter 2013-02-22, 21:39
+
Jay Kreps 2013-02-23, 04:13
+
Anthony Grimes 2013-02-22, 05:33