Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Help on running pig in local mode


Copy link to this message
-
Re: Help on running pig in local mode
lei tang 2012-10-19, 18:11
I checked the hadoop config and found that the staging dir specified does
not exist in local path (only in hdfs) and neither do I have write access.
So I changed the staging dir as below, but encounter some weird error
message:

[ltang01@stg-trgt00 ~]$ pig
-Dmapreduce.jobtracker.staging.root.dir=~/tmp/staging -x local
2012-10-19 11:06:49,624 [main] INFO  org.apache.pig.Main - Apache Pig
version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
2012-10-19 11:06:49,625 [main] INFO  org.apache.pig.Main - Logging error
messages to: /data1/home/ltang01/pig_1350670009621.log
2012-10-19 11:06:50,009 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: file:///
grunt> A = load 'toy.txt';
grunt> dump A;
2012-10-19 11:06:53,485 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2012-10-19 11:06:53,660 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2012-10-19 11:06:53,693 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2012-10-19 11:06:53,693 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2012-10-19 11:06:53,886 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2012-10-19 11:06:53,908 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2012-10-19 11:06:53,953 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2012-10-19 11:06:53,996 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
processName=JobTracker, sessionId2012-10-19 11:06:54,000 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2012-10-19 11:06:54,125 [Thread-5] WARN  org.apache.hadoop.mapred.JobClient
- No job jar file set.  User classes may not be found. See JobConf(Class)
or JobConf#setJar(String).
2012-10-19 11:06:54,207 [Thread-5] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2012-10-19 11:06:54,207 [Thread-5] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2012-10-19 11:06:54,222 [Thread-5] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2012-10-19 11:06:54,501 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2012-10-19 11:06:55,005 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_local_0001
2012-10-19 11:06:55,268 [Thread-6] INFO
org.apache.hadoop.mapreduce.util.ProcessTree - setsid exited with exit code
0
2012-10-19 11:06:55,824 [Thread-6] WARN
org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree -
/proc/<pid>/status does not have information about swap space used(VmSwap).
Can not track swap usage of a task.
2012-10-19 11:06:55,825 [Thread-6] INFO  org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@52cab854
2012-10-19 11:06:55,853 [Thread-6] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader
- Current split being processed file:/data1/home/ltang01/toy.txt:0+8
2012-10-19 11:06:55,914 [Thread-6] INFO  org.apache.hadoop.mapred.Task -
Task:attempt_local_0001_m_000000_0 is done. And is in the process of
commiting
2012-10-19 11:06:55,921 [Thread-6] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2012-10-19 11:06:55,922 [Thread-6] INFO  org.apache.hadoop.mapred.Task -
Task attempt_local_0001_m_000000_0 is allowed to commit now
2012-10-19 11:06:55,927 [Thread-6] INFO
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output
of task 'attempt_local_0001_m_000000_0' to
file:/tmp/temp487011820/tmp-1248641571
2012-10-19 11:06:55,927 [Thread-6] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2012-10-19 11:06:55,928 [Thread-6] INFO  org.apache.hadoop.mapred.Task -
Task 'attempt_local_0001_m_000000_0' done.
2012-10-19 11:06:55,929 [Thread-6] WARN
org.apache.hadoop.mapred.FileOutputCommitter - Output path is null in
cleanup
2012-10-19 11:06:56,380 [main] INFO  org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 0
time(s).
2012-10-19 11:06:57,382 [main] INFO  org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 1
time(s).
2012-10-19 11:06:58,383 [main] INFO  org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 2
time(s).
2012-10-19 11:06:59,385 [main] INFO  org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 3
time(s).
It keeps trying to connect to the local server. I have no idea what this is
about.  Can you help?

- Lei

On Wed, Oct 17, 2012 at 9:05 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote: