Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo, mail # user - Performance of table with large number of column families


Copy link to this message
-
Performance of table with large number of column families
Anthony Fox 2012-11-09, 16:26
I have a table set up to use the intersecting iterator pattern.  The
table has about 20M records which leads to 20M column families for the
data section - 1 unique column family per record.  The index section of
the table is not quite as large as the data section.  The rowkey is a
random padded integer partition between 0000000 and 9999999.  I turned
bloom filters on and used the ColumnFamilyFunctor to get performant
column family scans without specifying a range like in the bloom filter
examples in the README.  However, my column family scans (without any
custom iterator) are still fairly slow - ~30 seconds for a column family
batch scan of one record. I've also tried RowFunctor but I see similar
performance.  Can anyone shed any light on the performance metrics I'm
seeing?

Thanks,
Anthony
+
William Slacum 2012-11-09, 16:39
+
William Slacum 2012-11-09, 16:41
+
Anthony Fox 2012-11-09, 16:45
+
William Slacum 2012-11-09, 16:49
+
Anthony Fox 2012-11-09, 16:52
+
Anthony Fox 2012-11-09, 16:53
+
William Slacum 2012-11-09, 17:02
+
Anthony Fox 2012-11-09, 17:11
+
William Slacum 2012-11-09, 17:15
+
Anthony Fox 2012-11-09, 17:18
+
William Slacum 2012-11-09, 17:23
+
John Vines 2012-11-09, 17:41
+
Anthony Fox 2012-11-09, 18:02
+
John Vines 2012-11-09, 18:09
+
Anthony Fox 2012-11-09, 18:29
+
Eric Newton 2012-11-09, 18:32