Running a mixed 2 broker cluster. Mixed as in one of the broker1 is running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link. Directly using the tar ball, no local build used).
I have set the log.retention.minutes=10. However the broker is not honoring the setting. I see its not cleaning the log.dir at all.
However when I set the log.retention.hours=1, then it starts cleaning the log.
When I have the log.retention.minutes set in the server.properties then I see this logged in server.log:
Š.. [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not valid (kafka.utils.VerifiableProperties) [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not valid (kafka.utils.VerifiableProperties) ŠŠ I have set these properties too:
log.cleaner.enable=true log.cleanup.policy=delete But I see similar warning logged for these properties too.
I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0 and the other 0.8.1.1. Should the retention of topics for partitions owned/replicated by the broker running 0.8.1.1 not enforce the server properties settings as defined for that server.
So this brings an interesting question, in case of heterogeneous environment (as is in my case, which system parameters will take preference/precedence).
On 6/30/14, 9:19 AM, "Guozhang Wang" <[EMAIL PROTECTED]> wrote:
On a related note, in doing the upgrade from 0.8.0, I noticed that the config property changed from 'log.retention.hours' to 'log.retention.minutes'. Would it have made more sense to deprecate rather than replace there?
Also, I notice that internally, in the KafkaConfig class, it's represented as logRetentionTimeMillis() (e.g. not hours or minutes). And the per-topic version is in ms and not minutes. So, it all seems a bit confusing there (is there a reason for this)?
Jason On Tue, Jul 8, 2014 at 3:54 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
That's correct. The server where in I was running 0.8.1.1 was not honoring this parameter, despite the fact it was set in it server.properties. Not sure if this fact would play any role, the server which was running 0.8.0 was the leader for all the topics and partition in my setup. And the second server running 0.8.1.1 has all the replicas (follower).
On 7/8/14, 12:54 PM, "Guozhang Wang" <[EMAIL PROTECTED]> wrote:
Actually, Kafka only removes old segments. The last (active) segment is never removed. So, f you want to have a 10 min retention, you need to configure log rolling such that log segments are rolled at least every 10 mins.
Jun On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh < [EMAIL PROTECTED]lid> wrote:
Given that we now have support to log retention in minutes, I guess it would be apt to have rollup also have capability to be available in minutes. Whom/where should I ask to have that coded in.
One a similar note, I would genuinely want all the size and time based parameters to be defined at bytes and ms level. This would make it generic enough to have user choose and define kind of setting they want and not require someone to go and change the code to support a new use case. Having hours/minutes in config will just cause proliferation/bloating of config parameters. What is a good place to air this suggestion. Anyone?
On 7/9/14, 7:21 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
Has someone really used : log.retention.minutes in kafka 0.8.1.1. I have my full cluster running on 0.8.1.1 and the logs data is just not getting cleaned up.
And I see this message in kafka server.log ... [2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid (kafka.utils.VerifiableProperties) [2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid (kafka.utils.VerifiableProperties) ....
On 7/10/14, 10:07 AM, "Virendra Pratap Singh" <[EMAIL PROTECTED]LID> wrote:
No worries, its working. My deployment was flawed and it was running 0.8.0, while I was under the impression that its running 0.8.1.1. After going to proper 0.8.1.1, the parameter started working properly.
On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext