Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Help on running pig in local mode


Copy link to this message
-
Help on running pig in local mode
Hi,

I'm using Pig for my daily job. Pig dose a good job in mapreduce mode in
our internal hadoop cluster.  But it signals an error whenever I want to
run it in local mode.  I believe this is due to some permission or config
issue. Does anybody know how I can address this problem?  Any idea is
appreciated.

[ltang01@stg-trgt00 in]$ pig -x local
2012-10-11 17:29:25,707 [main] INFO  org.apache.pig.Main - Apache Pig
version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
2012-10-11 17:29:25,708 [main] INFO  org.apache.pig.Main - Logging error
messages to: /somewhere/in/pig_1350001765703.log
2012-10-11 17:29:26,068 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: file:///
grunt> A = load 'toy.txt';
grunt> dump A;
2012-10-11 17:29:36,056 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2012-10-11 17:29:36,218 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2012-10-11 17:29:36,249 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2012-10-11 17:29:36,250 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2012-10-11 17:29:36,382 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2012-10-11 17:29:36,403 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2012-10-11 17:29:36,445 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2012-10-11 17:29:36,486 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
processName=JobTracker, sessionId2012-10-11 17:29:36,493 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2012-10-11 17:29:36,995 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2012-10-11 17:29:37,002 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job null has failed! Stop running all dependent jobs
2012-10-11 17:29:37,002 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2012-10-11 17:29:37,007 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
recreate exception from backend error: ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
        at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
        at
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
        at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
        at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
        at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
        at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
        at java.lang.Thread.run(Thread.java:679)
        at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)

2012-10-11 17:29:37,007 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2012-10-11 17:29:37,007 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats
reported below may be incomplete
2012-10-11 17:29:37,009 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt
Features
0.20.2-dev      0.10.0  ltang01 2012-10-11 17:29:36     2012-10-11
17:29:37     UNKNOWN

Failed!

Failed Jobs:
JobId   Alias   Feature Message Outputs
N/A     A       MAP_ONLY        Message: ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
        at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
        at
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
        at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
        at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
        at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
        at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobCont
+
Prashant Kommireddi 2012-10-15, 22:45
+
lei tang 2012-10-15, 23:35
+
Dmitriy Ryaboy 2012-10-18, 04:05
+
lei tang 2012-10-19, 18:11
+
Dmitriy Ryaboy 2012-10-23, 00:39
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB