Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> auto-purge not purging


Copy link to this message
-
Re: auto-purge not purging
On Mon, Nov 4, 2013 at 7:46 AM, Ryan Fox <[EMAIL PROTECTED]> wrote:
>>
>> What version of ZooKeeper are you running? (solr 4.1.0 is pretty old,
>> 4.5.1 is out now, perhaps try upgrading?)
>>
>>

> Zookeeper version is 3.4.5.  This is part of a large software package, so
> upgrading the solr library would be painful, but possible.
>

3.4.5 is fine, I just wasn't sure what version solr was using, if it
was old it might be related but I don't think that's the case here.

>> "do not get autopurged" - could you provide more insight. 1) are you
>> sure the configuration options are being applied? 2) what does the
>> datadir look like? (snaps/logs, perhaps do an "ls -la" and provide in
>> this thread) 3) anything autopurge related in the zk server logs?
>>

>  Sure, the datadir contains only logs and snapshots.  I would expect the
> autoPurge function to delete logs and snapshots older than the number I
> have set in zoo.cfg, but nothing gets deleted, and they just pile up,
> eating up my hard drive.
> 1) The zookeeper instance is picking up my settings for clientPort, so I
> assume I have the purge settings in the correct place as well.  The
> relevant section of zoo.cfg:
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=1
>
> 2) ls -lah
> -rw-r--r--  1 ryanfox  staff    64M Oct 30 13:09 log.1
> -rw-r--r--  1 ryanfox  staff    64M Nov  4 09:36 log.48
> -rw-r--r--  1 ryanfox  staff    64M Nov  4 09:37 log.7b
> -rw-r--r--  1 ryanfox  staff   296B Oct 30 12:53 snapshot.0
> -rw-r--r--  1 ryanfox  staff    60K Nov  4 09:34 snapshot.47
> -rw-r--r--  1 ryanfox  staff    60K Nov  4 09:36 snapshot.7a
>

You have our snapRetainCount set to 3. That's why three snaps (and
their related logs) are saved.

Try setting it to 2.

> A new log and snapshot gets created every time I stop and start solr.  As
> an aside, every log file appears to be 64MB, no matter how much/little is
> in my index.  Is that correct?  Seems like a lot for an empty solr instance.
>

We pre-allocate the log files for performance. See "preAllocSize" here:
http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html

If you want smaller files try setting that to something like 10k or somesuch.

> 3) I haven't seen any smoking gun in the logs, but haven't specifically
> looked, either.  I'll let it go until it should have triggered the
> auto-purge and report back.

See if the above suggestions/info help.

Obviously don't carry over these changes to production. ;-)

Regards,

Patrick
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB