HBase, mail # dev - Experience with HBASE-8283 and lots of small hfile - 2014-04-04, 03:42
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
Experience with HBASE-8283 and lots of small hfile
  We are currently on 0.94.2(CDH 4.2.1) and would likely upgrade to 0.94.15 (CDH 4.6) primarily to use the above fix. We have turned off automatic major compactions. We load data into an hbase table every 2 minutes. Currently, we are not using bulk load since it created compaction issues. We noticed HBASE-8283 and could move to use this. Any gotchas on using this in production? Since, we could create a new HFile every 2 minutes, we would soon have a scenario where we would have a lot of hfiles. Would triggering a non major-compaction (using https://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#compact(byte[])) periodically be a reasonable compromise along with enabling Hbase-8283
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB