Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo >> mail # user >> Performance of table with large number of column families


Copy link to this message
-
Performance of table with large number of column families
I have a table set up to use the intersecting iterator pattern.  The
table has about 20M records which leads to 20M column families for the
data section - 1 unique column family per record.  The index section of
the table is not quite as large as the data section.  The rowkey is a
random padded integer partition between 0000000 and 9999999.  I turned
bloom filters on and used the ColumnFamilyFunctor to get performant
column family scans without specifying a range like in the bloom filter
examples in the README.  However, my column family scans (without any
custom iterator) are still fairly slow - ~30 seconds for a column family
batch scan of one record. I've also tried RowFunctor but I see similar
performance.  Can anyone shed any light on the performance metrics I'm
seeing?

Thanks,
Anthony
+
William Slacum 2012-11-09, 16:39
+
William Slacum 2012-11-09, 16:41
+
Anthony Fox 2012-11-09, 16:45
+
William Slacum 2012-11-09, 16:49
+
Anthony Fox 2012-11-09, 16:52
+
Anthony Fox 2012-11-09, 16:53
+
William Slacum 2012-11-09, 17:02
+
Anthony Fox 2012-11-09, 17:11
+
William Slacum 2012-11-09, 17:15
+
Anthony Fox 2012-11-09, 17:18
+
William Slacum 2012-11-09, 17:23
+
John Vines 2012-11-09, 17:41
+
Anthony Fox 2012-11-09, 18:02
+
John Vines 2012-11-09, 18:09
+
Anthony Fox 2012-11-09, 18:29
+
Eric Newton 2012-11-09, 18:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB