Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Map Join Problems


Copy link to this message
-
Map Join Problems
Using Hive 0.8.1 on Amazon EMR Hadoop Job.
Some problems with using mapjoin:
1) Exceed memory, I got the following errors.  Then I remove mapjoin in the query and instead set hive.auto.convert.join=true, thinking that let hive decides when mapjoin is suitable.  It does run much farther in the job, but then another similar error towards the end.
2) The I tried with mapjoin in the same query before and then set hive.mapjoin.localtask.max.memory.usage=3, same exact error.
My questions is that is there any other settings I can use to increase mapjoin memory or hashtable size?  Or is there any other better options?
2013-05-25 11:37:39 Starting to launch local task to process map join; maximum memory = 932118528
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/.versions/hive-0.8.1/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2013-05-25 11:37:57 Processing rows: 200000 Hashtable size: 199999 Memory usage: 776687416 rate: 0.833
2013-05-25 11:38:00 Processing rows: 215031 Hashtable size: 215031 Memory usage: 813018320 rate: 0.872
2013-05-25 11:38:00 Dump the hashtable into file: file:/tmp/hadoop/hive_2013-05-25_23-37-37_320_2027014861824847272/-local-10006/HashTable-Stage-6/MapJoin-bu-21--.hashtable
Execution failed with exit status: 2
Obtaining error information

Task failed!
Task ID:
  Stage-10

Logs:
3) Please look at the four errors example below.  My other question is that of all the runs with that mapjoin error, there is a pattern, the mapjoin is done on bu, which is a job before the error, all the error happens just shy of 4 rows of the bu table mapjoin, I found this too much of a coincidence, can someone please offer some insight?:#1:215035 Rows loaded to hdfs://10.190.182.26:9000/mnt/var/lib/hive_081/tmp/scratch/hive_2013-05-25_23-35-49_100_9059150281675034748/-ext-10000
MapReduce Jobs Launched:
Job 0: Map: 18  Reduce: 13   Accumulative CPU: 139.31 sec   HDFS Read: 1226414348 HDFS Write: 2179 SUCCESS
Job 1: Map: 9   Accumulative CPU: 54.1 sec   HDFS Read: 687306237 HDFS Write: 695722636 SUCCESS
Job 2: Map: 16   Accumulative CPU: 89.09 sec   HDFS Read: 695838641 HDFS Write: 703096594 SUCCESS
Total MapReduce CPU Time Spent: 4 minutes 42 seconds 500 msec
OK
Time taken: 108.206 seconds
OK
Time taken: 0.013 seconds
Total MapReduce jobs = 3
Execution log at: /tmp/hadoop/hadoop_20130525233737_8911fdca-6536-45bb-aac3-19b92b3de99c.log
2013-05-25 11:37:39 Starting to launch local task to process map join; maximum memory = 932118528
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/.versions/hive-0.8.1/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2013-05-25 11:37:57 Processing rows: 200000 Hashtable size: 199999 Memory usage: 776687416 rate: 0.833
2013-05-25 11:38:00 Processing rows: 215031 Hashtable size: 215031 Memory usage: 813018320 rate: 0.872
2013-05-25 11:38:00 Dump the hashtable into file: file:/tmp/hadoop/hive_2013-05-25_23-37-37_320_2027014861824847272/-local-10006/HashTable-Stage-6/MapJoin-bu-21--.hashtable
Execution failed with exit status: 2
Obtaining error information

Task failed!
Task ID:
  Stage-10

Logs:#2Table default.badurls stats: [num_partitions: 0, num_files: 18, num_rows: 0, total_size: 701922144, raw_data_size: 0]
214618 Rows loaded to hdfs://10.46.205.55:9000/mnt/var/lib/hive_081/tmp/scratch/hive_2013-05-25_23-12-54_513_8781300101638774300/-ext-10000
MapReduce Jobs Launched:
Job 0: Map: 21  Reduce: 13   Accumulative CPU: 142.11 sec   HDFS Read: 1225164183 HDFS Write: 2179 SUCCESS
Job 1: Map: 9   Accumulative CPU: 53.25 sec   HDFS Read: 686157231 HDFS Write: 694562725 SUCCESS
Job 2: Map: 18   Accumulative CPU: 92.26 sec   HDFS Read: 694650326 HDFS Write: 701922144 SUCCESS
Total MapReduce CPU Time Spent: 4 minutes 47 seconds 620 msec
OK
Time taken: 104.744 seconds
OK
Time taken: 0.013 seconds
Total MapReduce jobs = 3
Execution log at: /tmp/hadoop/hadoop_20130525231414_3cc50fdd-7e7a-4bcf-baab-b465804c6e49.log
2013-05-25 11:14:41 Starting to launch local task to process map join; maximum memory = 932118528
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/.versions/hive-0.8.1/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2013-05-25 11:15:04 Processing rows: 200000 Hashtable size: 199999 Memory usage: 776903128 rate: 0.833
2013-05-25 11:15:09 Processing rows: 214614 Hashtable size: 214614 Memory usage: 811592328 rate: 0.871
2013-05-25 11:15:09 Dump the hashtable into file: file:/tmp/hadoop/hive_2013-05-25_23-14-39_270_9001068894438685980/-local-10006/HashTable-Stage-6/MapJoin-bu-21--.hashtable
Execution failed with exit status: 2
Obtaining error information

Task failed!
Task ID:
  Stage-10

Logs:#3212590 Rows loaded to hdfs://10.96.162.172:9000/mnt/var/lib/hive_081/tmp/scratch/hive_2013-05-25_22-48-32_821_7905326830391538266/-ext-10000
MapReduce Jobs Launched:
Job 0: Map: 19  Reduce: 13   Accumulative CPU: 139.04 sec   HDFS Read: 1224727949 HDFS Write: 2179 SUCCESS
Job 1: Map: 9   Accumulative CPU: 53.03 sec   HDFS Read: 685799930 HDFS Write: 694158841 SUCCESS
Job 2: Map: 17   Accumulative CPU: 90.31 sec   HDFS Read: 694245117 HDFS Write: 701443582 SUCCESS
Total MapReduce CPU Time Spent: 4 minutes 42 seconds 380 msec
OK
Time taken: 105.029 seconds
OK
Time taken: 0.012 secon