Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Re: How to prevent major compaction when doing bulk load provisioning?


+
Jean-Daniel Cryans 2013-03-21, 18:03
+
ramkrishna vasudevan 2013-03-21, 18:05
+
Nicolas Seyvet 2013-03-21, 17:52
Copy link to this message
-
Re: How to prevent major compaction when doing bulk load provisioning?
@Ram: You are entirely correct, I made the exact same mistakes of mixing up
Large and minor compaction.  By looking closely, what I see is that at
around 200 HFiles per region it starts minor compacting files per group of
10 HFiles.  The "problem" seems that this minor compacting never stops even
when there are about 20 HFiles left.  It just keep on going and on taking
more and more time (I guess because the files to compact are getting
bigger).

Of course in parallel we keep on adding more and more data.

@J-D: "It seems to me that it would be better if you were able to do a
single load for all your files." Yes, I agree.. but that is not what we are
testing, our use case is to use 1min batch files.
+
Nicolas Seyvet 2013-03-22, 07:12
+
Jean-Daniel Cryans 2013-03-22, 16:32
+
Jean-Daniel Cryans 2013-03-21, 20:21
+
Ted Yu 2013-03-21, 20:05
+
Ted Yu 2013-03-21, 17:10
+
Amit Sela 2013-03-21, 16:47