Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Pig 0.8 HBaseStorage patch


Copy link to this message
-
Pig 0.8 HBaseStorage patch
Corbin Hoenes 2011-01-24, 21:22
We've got a patch we've made to HBaseStorage which allows a caller to turn
off the WriteAheadLog feature while doing bulk loads into hbase.

>From the performance tuning wikipage:
http://wiki.apache.org/hadoop/PerformanceTuning
"To speed up the inserts in a non critical job (like an import job), you can
use Put.writeToWAL(false) to bypass writing to the write ahead log."

We've tested this on HBase 0.20.6 and it helps dramatically.  It sounds like
future versions of HBase support a feature like this by default--so maybe
this problem goes away when we start using 0.90?

Is this something valuable to contribute back?