Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> map tasks are taking ever when running job on 24 TB


Copy link to this message
-
map tasks are taking ever when running job on 24 TB
Hi,

I'm running hive job on 24TB dataset (on 34560 partitions ). here about 500
to 1000 mappers are getting succeded (total of 80000) and rest mappaers are
taking for ever ( their status stays at 0% all times ).  Is there any
limitations on number of partitions/dataset ? are there any paraemeters to
set  here?

Same job  is suceeding on 18TB (25920 partitions ).

I already set below in my hive query.
set mapreduce.jobtracker.split.metainfo.maxsize=-1;
Regards,
Srinivas
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB