Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hive dynamic partition error?


Copy link to this message
-
Hive dynamic partition error?
So we're seeing the following error during some of our hive loads:

2011-07-05 12:26:52,927 Stage-2 map = 100%,  reduce = 100%
Ended Job = job_201106302113_3864
Loading data to table default.merged_weblogs partition (day=null)
Failed with exception Number of dynamic partitions created is 1013,
which is more than 1000. To solve this try to set
hive.exec.max.dynamic.partitions to at least 1013.
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask

Here is a sample script we're running:

SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.max.dynamic.partitions.pernode=10000;
SET hive.exec.max.dynamic.partitions=10000;
SET hive.exec.max.created.files=150000;

SET hive.exec.compress.intermediate=true;
SET hive.intermediate.compression.codec=com.hadoop.compression.lzo.LzoCodec;
SET hive.intermediate.compression.type=BLOCK;
SET mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;

SET hive.exec.compress.output=true;
SET mapred.output.compress=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
SET mapred.output.compression.type=BLOCK;

FROM (
   SELECT hostname, name, ip, day
   FROM logsStaging
   UNION ALL
   SELECT hostname, name, ip, day
   FROM logs
) a

INSERT OVERWRITE TABLE logs PARTITION(day)
SELECT DISTINCT hostname, name, ip, day
DISTRIBUTE BY day;

QUIT;
Has anyone run into this problem before?  And I've noticed that
increasing the number of partitions hasn't been working.  Been looking
for the config.xml setting already configured with 'final' in the
properties but no go so far.  I believe the default is 100 partitions
and the job (when running) does show 10000 partitions (from the above
script)

thoughts on what else to look at?

Thanks!