Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Why BucketJoinMap consume too much memory


+
binhnt22 2012-03-31, 01:16
+
Bejoy Ks 2012-04-01, 19:35
+
Amit Sharma 2012-04-03, 17:36
+
Bejoy Ks 2012-04-05, 08:07
+
binhnt22 2012-04-05, 10:07
+
Nitin Pawar 2012-04-05, 10:36
+
binhnt22 2012-04-05, 10:52
Copy link to this message
-
Re: Why BucketJoinMap consume too much memory
Can you tell me the size of table b?

If you are doing bucketing and still size b table is huge then it will
reach this problem

On Thu, Apr 5, 2012 at 4:22 PM, binhnt22 <[EMAIL PROTECTED]> wrote:

>  Thank Nitin,****
>
> ** **
>
> I tried but no luck. Here’s hive log, please spend a little time to view
> it.****
>
> ** **
>
> hive> set hive.optimize.bucketmapjoin = true;****
>
> hive> set hive.enforce.bucketing=true;****
>
> hive> set
> hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;****
>
> hive> select /*+ MAPJOIN(b) */ * from ra_md_syn a join ra_ocs_syn b****
>
>     > on (a.calling = b.calling) where  a.total_volume <> b.total_volume;*
> ***
>
> Total MapReduce jobs = 1****
>
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please
> use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties
> files.****
>
> Execution log at:
> /tmp/hduser/hduser_20120405103737_28ef26fe-a202-4047-b5ca-c40d9e3ad36c.log
> ****
>
> 2012-04-05 10:37:45     Starting to launch local task to process map
> join;      maximum memory = 1398145024****
>
> 2012-04-05 10:37:48     Processing rows:        200000  Hashtable size:
> 199999  Memory usage:   75403880        rate:   0.054****
>
> 2012-04-05 10:37:50     Processing rows:        300000  Hashtable size:
> 299999  Memory usage:   111404664       rate:   0.08****
>
> 2012-04-05 10:37:54     Processing rows:        400000  Hashtable size:
> 399999  Memory usage:   151598960       rate:   0.108****
>
> 2012-04-05 10:38:04     Processing rows:        500000  Hashtable size:
> 499999  Memory usage:   185483368       rate:   0.133****
>
> 2012-04-05 10:38:09     Processing rows:        600000  Hashtable size:
> 599999  Memory usage:   221483392       rate:   0.158****
>
> 2012-04-05 10:38:13     Processing rows:        700000  Hashtable size:
> 699999  Memory usage:   257482640       rate:   0.184****
>
> 2012-04-05 10:38:19     Processing rows:        800000  Hashtable size:
> 799999  Memory usage:   297676944       rate:   0.213****
>
> 2012-04-05 10:38:22     Processing rows:        900000  Hashtable size:
> 899999  Memory usage:   333676968       rate:   0.239****
>
> 2012-04-05 10:38:27     Processing rows:        1000000 Hashtable size:
> 999999  Memory usage:   369676944       rate:   0.264****
>
> 2012-04-05 10:38:31     Processing rows:        1100000 Hashtable size:
> 1099999 Memory usage:   405676968       rate:   0.29****
>
> 2012-04-05 10:38:36     Processing rows:        1200000 Hashtable size:
> 1199999 Memory usage:   441676944       rate:   0.316****
>
> 2012-04-05 10:38:42     Processing rows:        1300000 Hashtable size:
> 1299999 Memory usage:   477676944       rate:   0.342****
>
> 2012-04-05 10:38:47     Processing rows:        1400000 Hashtable size:
> 1399999 Memory usage:   513676968       rate:   0.367****
>
> 2012-04-05 10:38:52     Processing rows:        1500000 Hashtable size:
> 1499999 Memory usage:   549676944       rate:   0.393****
>
> 2012-04-05 10:39:00     Processing rows:        1600000 Hashtable size:
> 1599999 Memory usage:   602454200       rate:   0.431****
>
> 2012-04-05 10:39:08     Processing rows:        1700000 Hashtable size:
> 1699999 Memory usage:   630065552       rate:   0.451****
>
> 2012-04-05 10:39:14     Processing rows:        1800000 Hashtable size:
> 1799999 Memory usage:   666065552       rate:   0.476****
>
> 2012-04-05 10:39:20     Processing rows:        1900000 Hashtable size:
> 1899999 Memory usage:   702065552       rate:   0.502****
>
> 2012-04-05 10:39:26     Processing rows:        2000000 Hashtable size:
> 1999999 Memory usage:   738065576       rate:   0.528****
>
> 2012-04-05 10:39:36     Processing rows:        2100000 Hashtable size:
> 2099999 Memory usage:   774065552       rate:   0.554****
>
> 2012-04-05 10:39:43     Processing rows:        2200000 Hashtable size:
> 2199999 Memory usage:   810065552       rate:   0.579****
>
> 2012-04-05 10:39:51     Processing rows:        2300000 Hashtable size:

Nitin Pawar
+
Bejoy Ks 2012-04-05, 12:22
+
binhnt22 2012-04-06, 01:19
+
gemini alex 2012-04-06, 07:06
+
Bejoy Ks 2012-04-06, 16:33
+
binhnt22 2012-04-09, 03:12
+
Bejoy Ks 2012-04-09, 14:48
+
binhnt22 2012-04-10, 02:40
+
Bejoy Ks 2012-04-10, 15:43