Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Fine tunning


Copy link to this message
-
Re: Fine tunning
Michael Segel 2014-01-17, 12:19
In your mapreduce, in the mapper.setup() read the lookup tables in to memory and then access them as needed in your Mapper.map() method.
Same for reducer...

See mapper  joins in Map/Reduce ...

On Jan 6, 2014, at 3:23 AM, Ranjini Rathinam <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have a input File of 16 fields in it.
>
> Using Mapreduce code need to load the hbase tables.
>
> The first eight has to go into one table in hbase and last eight has to got
> to another hbase table.
>
> The data is being loaded into hbase table in 0.11 sec , but if any lookup
> is being added in the mapreduce code,
> For eg, the input file has one  attribute named currency , it will have a
> master table currency. need to match both values to print it.
>
> The table which has lookup takes long time to get load. For 13250 records
> it take 59 mins.
>
> How to make fine tune to reduce the time for its loading.
>
> Please help.
>
> Thanks in advance.
>
> Ranjini.R

The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com