Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> hive mapred problem


Copy link to this message
-
RE: hive mapred problem
Hi Nitin,

Job tracker log:-->

2012-10-08 15:43:29,331 WARN org.apache.hadoop.conf.Configuration: /usr/local/hadoopstorage/mapred/local/jobTracker/job_201210051717_0015.xml:a attempt to override fina

l parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:29,331 WARN org.apache.hadoop.conf.Configuration: /usr/local/hadoopstorage/mapred/local/jobTracker/job_201210051717_0015.xml:a attempt to override fina

l parameter: mapred.map.tasks;  Ignoring.

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobInProgress: job_201210051717_0015: nMaps=10 nReduces=2 max=-1

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobTracker: Job job_201210051717_0015 added successfully for user 'hadoop' to queue 'default'

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.AuditLogger: USER=hadoop  IP=10.99.42.9   OPERATION=SUBMIT_JOB    TARGET=job_201210051717_0015    RESULT=SUCCESS

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201210051717_0015

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201210051717_0015

2012-10-08 15:43:29,350 INFO org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with users keys in /usr/local/hadoopstorage/mapred/system/job_2012100

51717_0015/jobToken

2012-10-08 15:43:29,352 ERROR org.apache.hadoop.mapred.JobTracker: Job initialization failed:

java.io.IOException: Number of maps in JobConf doesn't match number of recieved splits for job job_201210051717_0015! numMapTasks=10, #splits=1

        at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:703)

        at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207)

        at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:662)

2012-10-08 15:43:29,352 INFO org.apache.hadoop.mapred.JobTracker: Failing job job_201210051717_0015

2012-10-08 15:43:29,352 INFO org.apache.hadoop.mapred.JobInProgress$JobSummary: jobId=job_201210051717_0015,submitTime=1349691209316,launchTime=0,,finishTime=1349691209

352,numMaps=0,numSlotsPerMap=1,numReduces=0,numSlotsPerReduce=1,user=hadoop,queue=default,status=FAILED,mapSlotSeconds=0,reduceSlotsSeconds=0,clusterMapCapacity=14,clus

terReduceCapacity=14,jobName=select count(1) from pokes(Stage-1)

2012-10-08 15:43:29,353 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/history/job_201210051717_0015_1349691209316_hadoop_select+count%281%29+fro

m+pokes%28Stage-1%29 to file:/var/log/hadoop/history/done/version-1/NHCLT-PC44-2_1349437647390_/2012/10/08/000000

2012-10-08 15:43:29,356 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/history/job_201210051717_0015_conf.xml to file:/var/log/hadoop/history/don

e/version-1/NHCLT-PC44-2_1349437647390_/2012/10/08/000000

2012-10-08 15:43:30,362 INFO org.apache.hadoop.mapred.JobTracker: Killing job job_201210051717_0015

2012-10-08 17:17:35,641 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation

tokens

and hive log:-->

2012-10-08 15:43:08,597 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:08,606 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:08,606 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,000 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,004 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,004 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,014 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,018 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:09,018 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:17,309 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:17,319 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:17,320 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:17,759 WARN  conf.Configuration (Configuration.java:loadResource(1245)) - file:/tmp/hive-default-74584299235925841