Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> hive mapred problem


Copy link to this message
-
RE: hive mapred problem
Hi Nitin,

Job tracker log:-->

2012-10-08 15:43:29,331 WARN org.apache.hadoop.conf.Configuration: /usr/local/hadoopstorage/mapred/local/jobTracker/job_201210051717_0015.xml:a attempt to override fina

l parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:29,331 WARN org.apache.hadoop.conf.Configuration: /usr/local/hadoopstorage/mapred/local/jobTracker/job_201210051717_0015.xml:a attempt to override fina

l parameter: mapred.map.tasks;  Ignoring.

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobInProgress: job_201210051717_0015: nMaps=10 nReduces=2 max=-1

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobTracker: Job job_201210051717_0015 added successfully for user 'hadoop' to queue 'default'

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.AuditLogger: USER=hadoop  IP=10.99.42.9   OPERATION=SUBMIT_JOB    TARGET=job_201210051717_0015    RESULT=SUCCESS

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201210051717_0015

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201210051717_0015

2012-10-08 15:43:29,350 INFO org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with users keys in /usr/local/hadoopstorage/mapred/system/job_2012100

51717_0015/jobToken

2012-10-08 15:43:29,352 ERROR org.apache.hadoop.mapred.JobTracker: Job initialization failed:

java.io.IOException: Number of maps in JobConf doesn't match number of recieved splits for job job_201210051717_0015! numMapTasks=10, #splits=1

        at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:703)

        at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207)

        at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:662)

2012-10-08 15:43:29,352 INFO org.apache.hadoop.mapred.JobTracker: Failing job job_201210051717_0015

2012-10-08 15:43:29,352 INFO org.apache.hadoop.mapred.JobInProgress$JobSummary: jobId=job_201210051717_0015,submitTime=1349691209316,launchTime=0,,finishTime=1349691209

352,numMaps=0,numSlotsPerMap=1,numReduces=0,numSlotsPerReduce=1,user=hadoop,queue=default,status=FAILED,mapSlotSeconds=0,reduceSlotsSeconds=0,clusterMapCapacity=14,clus

terReduceCapacity=14,jobName=select count(1) from pokes(Stage-1)

2012-10-08 15:43:29,353 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/history/job_201210051717_0015_1349691209316_hadoop_select+count%281%29+fro

m+pokes%28Stage-1%29 to file:/var/log/hadoop/history/done/version-1/NHCLT-PC44-2_1349437647390_/2012/10/08/000000

2012-10-08 15:43:29,356 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/history/job_201210051717_0015_conf.xml to file:/var/log/hadoop/history/don

e/version-1/NHCLT-PC44-2_1349437647390_/2012/10/08/000000

2012-10-08 15:43:30,362 INFO org.apache.hadoop.mapred.JobTracker: Killing job job_201210051717_0015

2012-10-08 17:17:35,641 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation

tokens

and hive log:-->

2012-10-08 15:43:08,597 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:08,606 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:08,606 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,000 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,004 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,004 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,014 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,018 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,018 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:17,309 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:17,319 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:17,320 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:17,759 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/tmp/hive-default-74584299235925841
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB