Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Increase number of reducers for bulk data load to empty HBase table


Copy link to this message
-
Re: Increase number of reducers for bulk data load to empty HBase table
Hey Matthew,

The only way to increase the number of reducers is to have more regions -
each reducer produces an output per region, so the number of reducers =number of regions.

Thanks
Karthik
On 10/18/11 2:00 AM, "Matthew Tovbin" <[EMAIL PROTECTED]> wrote:

>Hello, Guys,
>
>I'm willing to bulk load data from hdfs folders into HBase, for this
>purpose
>I used configureIncrementalLoad method from HFileOutputFormat that
>configures the job, as follows:
>
>org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLo
>ad(job,
>myTable)
>
>The problem is that destination table in HBase is empty, meaning it's only
>hosted by one region server, so the resulted number of reducers is 1,
>which
>makes the job to run almost forever.
>
>How can I increase the number of reducers? Can the number of reducers be
>set
>to more than a number of region servers?
>
>Thanks in advance,
>     Matthew Tovbin.