Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Increase number of reducers for bulk data load to empty HBase table


Copy link to this message
-
Re: Increase number of reducers for bulk data load to empty HBase table
Hey Matthew,

The only way to increase the number of reducers is to have more regions -
each reducer produces an output per region, so the number of reducers =number of regions.

Thanks
Karthik
On 10/18/11 2:00 AM, "Matthew Tovbin" <[EMAIL PROTECTED]> wrote:

>Hello, Guys,
>
>I'm willing to bulk load data from hdfs folders into HBase, for this
>purpose
>I used configureIncrementalLoad method from HFileOutputFormat that
>configures the job, as follows:
>
>org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLo
>ad(job,
>myTable)
>
>The problem is that destination table in HBase is empty, meaning it's only
>hosted by one region server, so the resulted number of reducers is 1,
>which
>makes the job to run almost forever.
>
>How can I increase the number of reducers? Can the number of reducers be
>set
>to more than a number of region servers?
>
>Thanks in advance,
>     Matthew Tovbin.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB