Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> auto-purge not purging


Copy link to this message
-
Re: auto-purge not purging
Ah, sorry I wasn't clear.  There's currently 3 log/snapshot pairs in that
dir, because I cleared it out recently.  If I keep starting and stopping
solr/zookeeper, they'll continue to pile up.

$ ls -lah

drwxr-xr-x  14 ryanfox  staff   476B Nov  4 17:25 .
drwxr-xr-x   4 ryanfox  staff   136B Nov  4 09:30 ..
-rw-r--r--   1 ryanfox  staff    64M Oct 30 13:09 log.1
-rw-r--r--   1 ryanfox  staff    64M Nov  4 17:25 log.112
-rw-r--r--   1 ryanfox  staff    64M Nov  4 09:36 log.48
-rw-r--r--   1 ryanfox  staff    64M Nov  4 17:22 log.7b
-rw-r--r--   1 ryanfox  staff    64M Nov  4 17:22 log.ad
-rw-r--r--   1 ryanfox  staff    64M Nov  4 17:25 log.e1
-rw-r--r--   1 ryanfox  staff   296B Oct 30 12:53 snapshot.0
-rw-r--r--   1 ryanfox  staff    60K Nov  4 17:25 snapshot.111
-rw-r--r--   1 ryanfox  staff    60K Nov  4 09:34 snapshot.47
-rw-r--r--   1 ryanfox  staff    60K Nov  4 09:36 snapshot.7a
-rw-r--r--   1 ryanfox  staff    60K Nov  4 17:22 snapshot.ac
-rw-r--r--   1 ryanfox  staff    60K Nov  4 17:23 snapshot.e0

Thanks for the info on preAllocSize, that may prove useful as well.
Nothing showed up in the logs today despite running for ~8 hours.

Best,
Ryan
On Mon, Nov 4, 2013 at 5:13 PM, Patrick Hunt <[EMAIL PROTECTED]> wrote:

> On Mon, Nov 4, 2013 at 7:46 AM, Ryan Fox <[EMAIL PROTECTED]> wrote:
> >>
> >> What version of ZooKeeper are you running? (solr 4.1.0 is pretty old,
> >> 4.5.1 is out now, perhaps try upgrading?)
> >>
> >>
>
> > Zookeeper version is 3.4.5.  This is part of a large software package, so
> > upgrading the solr library would be painful, but possible.
> >
>
> 3.4.5 is fine, I just wasn't sure what version solr was using, if it
> was old it might be related but I don't think that's the case here.
>
> >> "do not get autopurged" - could you provide more insight. 1) are you
> >> sure the configuration options are being applied? 2) what does the
> >> datadir look like? (snaps/logs, perhaps do an "ls -la" and provide in
> >> this thread) 3) anything autopurge related in the zk server logs?
> >>
>
> >  Sure, the datadir contains only logs and snapshots.  I would expect the
> > autoPurge function to delete logs and snapshots older than the number I
> > have set in zoo.cfg, but nothing gets deleted, and they just pile up,
> > eating up my hard drive.
> > 1) The zookeeper instance is picking up my settings for clientPort, so I
> > assume I have the purge settings in the correct place as well.  The
> > relevant section of zoo.cfg:
> > autopurge.snapRetainCount=3
> > autopurge.purgeInterval=1
> >
> > 2) ls -lah
> > -rw-r--r--  1 ryanfox  staff    64M Oct 30 13:09 log.1
> > -rw-r--r--  1 ryanfox  staff    64M Nov  4 09:36 log.48
> > -rw-r--r--  1 ryanfox  staff    64M Nov  4 09:37 log.7b
> > -rw-r--r--  1 ryanfox  staff   296B Oct 30 12:53 snapshot.0
> > -rw-r--r--  1 ryanfox  staff    60K Nov  4 09:34 snapshot.47
> > -rw-r--r--  1 ryanfox  staff    60K Nov  4 09:36 snapshot.7a
> >
>
> You have our snapRetainCount set to 3. That's why three snaps (and
> their related logs) are saved.
>
> Try setting it to 2.
>
> > A new log and snapshot gets created every time I stop and start solr.  As
> > an aside, every log file appears to be 64MB, no matter how much/little is
> > in my index.  Is that correct?  Seems like a lot for an empty solr
> instance.
> >
>
> We pre-allocate the log files for performance. See "preAllocSize" here:
> http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html
>
> If you want smaller files try setting that to something like 10k or
> somesuch.
>
> > 3) I haven't seen any smoking gun in the logs, but haven't specifically
> > looked, either.  I'll let it go until it should have triggered the
> > auto-purge and report back.
>
> See if the above suggestions/info help.
>
> Obviously don't carry over these changes to production. ;-)
>
> Regards,
>
> Patrick
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB