Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Help on running pig in local mode


Copy link to this message
-
Help on running pig in local mode
lei tang 2012-10-15, 22:43
Hi,

I'm using Pig for my daily job. Pig dose a good job in mapreduce mode in
our internal hadoop cluster.  But it signals an error whenever I want to
run it in local mode.  I believe this is due to some permission or config
issue. Does anybody know how I can address this problem?  Any idea is
appreciated.

[ltang01@stg-trgt00 in]$ pig -x local
2012-10-11 17:29:25,707 [main] INFO  org.apache.pig.Main - Apache Pig
version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
2012-10-11 17:29:25,708 [main] INFO  org.apache.pig.Main - Logging error
messages to: /somewhere/in/pig_1350001765703.log
2012-10-11 17:29:26,068 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: file:///
grunt> A = load 'toy.txt';
grunt> dump A;
2012-10-11 17:29:36,056 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2012-10-11 17:29:36,218 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2012-10-11 17:29:36,249 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2012-10-11 17:29:36,250 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2012-10-11 17:29:36,382 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2012-10-11 17:29:36,403 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2012-10-11 17:29:36,445 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2012-10-11 17:29:36,486 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
processName=JobTracker, sessionId2012-10-11 17:29:36,493 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2012-10-11 17:29:36,995 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2012-10-11 17:29:37,002 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job null has failed! Stop running all dependent jobs
2012-10-11 17:29:37,002 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2012-10-11 17:29:37,007 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
recreate exception from backend error: ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
        at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
        at
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
        at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
        at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
        at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
        at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
        at java.lang.Thread.run(Thread.java:679)
        at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)

2012-10-11 17:29:37,007 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2012-10-11 17:29:37,007 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats
reported below may be incomplete
2012-10-11 17:29:37,009 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt
Features
0.20.2-dev      0.10.0  ltang01 2012-10-11 17:29:36     2012-10-11
17:29:37     UNKNOWN

Failed!

Failed Jobs:
JobId   Alias   Feature Message Outputs
N/A     A       MAP_ONLY        Message: ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
        at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
        at
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
        at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
        at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
        at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
        at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobCont
+
Prashant Kommireddi 2012-10-15, 22:45
+
lei tang 2012-10-15, 23:35
+
Dmitriy Ryaboy 2012-10-18, 04:05
+
lei tang 2012-10-19, 18:11
+
Dmitriy Ryaboy 2012-10-23, 00:39