Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Why BucketJoinMap consume too much memory


+
binhnt22 2012-03-31, 01:16
+
Bejoy Ks 2012-04-01, 19:35
+
Amit Sharma 2012-04-03, 17:36
+
Bejoy Ks 2012-04-05, 08:07
+
binhnt22 2012-04-05, 10:07
+
Nitin Pawar 2012-04-05, 10:36
+
binhnt22 2012-04-05, 10:52
Copy link to this message
-
Re: Why BucketJoinMap consume too much memory
Can you tell me the size of table b?

If you are doing bucketing and still size b table is huge then it will
reach this problem

On Thu, Apr 5, 2012 at 4:22 PM, binhnt22 <[EMAIL PROTECTED]> wrote:

>  Thank Nitin,****
>
> ** **
>
> I tried but no luck. Here’s hive log, please spend a little time to view
> it.****
>
> ** **
>
> hive> set hive.optimize.bucketmapjoin = true;****
>
> hive> set hive.enforce.bucketing=true;****
>
> hive> set
> hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;****
>
> hive> select /*+ MAPJOIN(b) */ * from ra_md_syn a join ra_ocs_syn b****
>
>     > on (a.calling = b.calling) where  a.total_volume <> b.total_volume;*
> ***
>
> Total MapReduce jobs = 1****
>
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please
> use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties
> files.****
>
> Execution log at:
> /tmp/hduser/hduser_20120405103737_28ef26fe-a202-4047-b5ca-c40d9e3ad36c.log
> ****
>
> 2012-04-05 10:37:45     Starting to launch local task to process map
> join;      maximum memory = 1398145024****
>
> 2012-04-05 10:37:48     Processing rows:        200000  Hashtable size:
> 199999  Memory usage:   75403880        rate:   0.054****
>
> 2012-04-05 10:37:50     Processing rows:        300000  Hashtable size:
> 299999  Memory usage:   111404664       rate:   0.08****
>
> 2012-04-05 10:37:54     Processing rows:        400000  Hashtable size:
> 399999  Memory usage:   151598960       rate:   0.108****
>
> 2012-04-05 10:38:04     Processing rows:        500000  Hashtable size:
> 499999  Memory usage:   185483368       rate:   0.133****
>
> 2012-04-05 10:38:09     Processing rows:        600000  Hashtable size:
> 599999  Memory usage:   221483392       rate:   0.158****
>
> 2012-04-05 10:38:13     Processing rows:        700000  Hashtable size:
> 699999  Memory usage:   257482640       rate:   0.184****
>
> 2012-04-05 10:38:19     Processing rows:        800000  Hashtable size:
> 799999  Memory usage:   297676944       rate:   0.213****
>
> 2012-04-05 10:38:22     Processing rows:        900000  Hashtable size:
> 899999  Memory usage:   333676968       rate:   0.239****
>
> 2012-04-05 10:38:27     Processing rows:        1000000 Hashtable size:
> 999999  Memory usage:   369676944       rate:   0.264****
>
> 2012-04-05 10:38:31     Processing rows:        1100000 Hashtable size:
> 1099999 Memory usage:   405676968       rate:   0.29****
>
> 2012-04-05 10:38:36     Processing rows:        1200000 Hashtable size:
> 1199999 Memory usage:   441676944       rate:   0.316****
>
> 2012-04-05 10:38:42     Processing rows:        1300000 Hashtable size:
> 1299999 Memory usage:   477676944       rate:   0.342****
>
> 2012-04-05 10:38:47     Processing rows:        1400000 Hashtable size:
> 1399999 Memory usage:   513676968       rate:   0.367****
>
> 2012-04-05 10:38:52     Processing rows:        1500000 Hashtable size:
> 1499999 Memory usage:   549676944       rate:   0.393****
>
> 2012-04-05 10:39:00     Processing rows:        1600000 Hashtable size:
> 1599999 Memory usage:   602454200       rate:   0.431****
>
> 2012-04-05 10:39:08     Processing rows:        1700000 Hashtable size:
> 1699999 Memory usage:   630065552       rate:   0.451****
>
> 2012-04-05 10:39:14     Processing rows:        1800000 Hashtable size:
> 1799999 Memory usage:   666065552       rate:   0.476****
>
> 2012-04-05 10:39:20     Processing rows:        1900000 Hashtable size:
> 1899999 Memory usage:   702065552       rate:   0.502****
>
> 2012-04-05 10:39:26     Processing rows:        2000000 Hashtable size:
> 1999999 Memory usage:   738065576       rate:   0.528****
>
> 2012-04-05 10:39:36     Processing rows:        2100000 Hashtable size:
> 2099999 Memory usage:   774065552       rate:   0.554****
>
> 2012-04-05 10:39:43     Processing rows:        2200000 Hashtable size:
> 2199999 Memory usage:   810065552       rate:   0.579****
>
> 2012-04-05 10:39:51     Processing rows:        2300000 Hashtable size:

Nitin Pawar
+
Bejoy Ks 2012-04-05, 12:22
+
binhnt22 2012-04-06, 01:19
+
gemini alex 2012-04-06, 07:06
+
Bejoy Ks 2012-04-06, 16:33
+
binhnt22 2012-04-09, 03:12
+
Bejoy Ks 2012-04-09, 14:48
+
binhnt22 2012-04-10, 02:40
+
Bejoy Ks 2012-04-10, 15:43
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB