-
Re: Help on running pig in local modePrashant Kommireddi 2012-10-15, 22:45
Is this file present on your local FS?
file:///data1/home/ltang01/mf/prep/in/toy.txt
On Mon, Oct 15, 2012 at 3:43 PM, lei tang <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I'm using Pig for my daily job. Pig dose a good job in mapreduce mode in
> our internal hadoop cluster. But it signals an error whenever I want to
> run it in local mode. I believe this is due to some permission or config
> issue. Does anybody know how I can address this problem? Any idea is
> appreciated.
>
> [ltang01@stg-trgt00 in]$ pig -x local
> 2012-10-11 17:29:25,707 [main] INFO org.apache.pig.Main - Apache Pig
> version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
> 2012-10-11 17:29:25,708 [main] INFO org.apache.pig.Main - Logging error
> messages to: /somewhere/in/pig_1350001765703.log
> 2012-10-11 17:29:26,068 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: file:///
> grunt> A = load 'toy.txt';
> grunt> dump A;
> 2012-10-11 17:29:36,056 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2012-10-11 17:29:36,218 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2012-10-11 17:29:36,249 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2012-10-11 17:29:36,250 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2012-10-11 17:29:36,382 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to the job
> 2012-10-11 17:29:36,403 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2012-10-11 17:29:36,445 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2012-10-11 17:29:36,486 [main] INFO
> org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
> processName=JobTracker, sessionId> 2012-10-11 17:29:36,493 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2012-10-11 17:29:36,995 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2012-10-11 17:29:37,002 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job null has failed! Stop running all dependent jobs
> 2012-10-11 17:29:37,002 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> 2012-10-11 17:29:37,007 [main] ERROR
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> recreate exception from backend error: ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at
>
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
> at
> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
> at
>
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
file:///data1/home/ltang01/mf/prep/in/toy.txt
On Mon, Oct 15, 2012 at 3:43 PM, lei tang <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I'm using Pig for my daily job. Pig dose a good job in mapreduce mode in
> our internal hadoop cluster. But it signals an error whenever I want to
> run it in local mode. I believe this is due to some permission or config
> issue. Does anybody know how I can address this problem? Any idea is
> appreciated.
>
> [ltang01@stg-trgt00 in]$ pig -x local
> 2012-10-11 17:29:25,707 [main] INFO org.apache.pig.Main - Apache Pig
> version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
> 2012-10-11 17:29:25,708 [main] INFO org.apache.pig.Main - Logging error
> messages to: /somewhere/in/pig_1350001765703.log
> 2012-10-11 17:29:26,068 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: file:///
> grunt> A = load 'toy.txt';
> grunt> dump A;
> 2012-10-11 17:29:36,056 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2012-10-11 17:29:36,218 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2012-10-11 17:29:36,249 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2012-10-11 17:29:36,250 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2012-10-11 17:29:36,382 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to the job
> 2012-10-11 17:29:36,403 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2012-10-11 17:29:36,445 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2012-10-11 17:29:36,486 [main] INFO
> org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
> processName=JobTracker, sessionId> 2012-10-11 17:29:36,493 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2012-10-11 17:29:36,995 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2012-10-11 17:29:37,002 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job null has failed! Stop running all dependent jobs
> 2012-10-11 17:29:37,002 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> 2012-10-11 17:29:37,007 [main] ERROR
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> recreate exception from backend error: ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at
>
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
> at
> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
> at
>
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
