I've upgraded to 0.94.11. Here is my "worst-case scenario" :
- let say each regionserver has 3 GB memstore
- let say compaction max filesize is ~200 GB, min. 2 files, max 10 files.
- let say memstore is growing "slowly" (1 GB / hour per RS)
Then, automatically flushing every hour will lead into 1 GB storefiles,
being compacted into storefiles of 2 GB, 3 GB, 4.... up to 200 GB.
Sometimes, my write-load becomes very low, and periodic flusher will flush
perhaps 1 MB of data, it will trigger a minor compaction of hundreds
gigabytes + 1 MB; it seems to be lots of IO just to merge 1 MB of data.
Previously (ie. lack of periodic flusher) memstore was creating 3 GB
storefiles, and thus creating (after minor compactions) 3 GB, 6 GB, 9 GB...
up to 200 GB storefiles. And if memstore is growing slowly, it won't
generate small storefiles on HDFS. If think it looks like a more reasonable
IO-load, doesn't it?
I deeply agree with Periodic Flusher relevance, but I don't think it's
suitable for everyone. Do you share my opinion wrt. my workload?
On Sun, Dec 8, 2013 at 10:36 PM, Ted Yu <[EMAIL PROTECTED]> wrote: