Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Fine tunning


Copy link to this message
-
Re: Fine tunning
In your mapreduce, in the mapper.setup() read the lookup tables in to memory and then access them as needed in your Mapper.map() method.
Same for reducer...

See mapper  joins in Map/Reduce ...

On Jan 6, 2014, at 3:23 AM, Ranjini Rathinam <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have a input File of 16 fields in it.
>
> Using Mapreduce code need to load the hbase tables.
>
> The first eight has to go into one table in hbase and last eight has to got
> to another hbase table.
>
> The data is being loaded into hbase table in 0.11 sec , but if any lookup
> is being added in the mapreduce code,
> For eg, the input file has one  attribute named currency , it will have a
> master table currency. need to match both values to print it.
>
> The table which has lookup takes long time to get load. For 13250 records
> it take 59 mins.
>
> How to make fine tune to reduce the time for its loading.
>
> Please help.
>
> Thanks in advance.
>
> Ranjini.R

The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB