Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Help on running pig in local mode


Copy link to this message
-
Re: Help on running pig in local mode
I checked the hadoop config and found that the staging dir specified does
not exist in local path (only in hdfs) and neither do I have write access.
So I changed the staging dir as below, but encounter some weird error
message:

[ltang01@stg-trgt00 ~]$ pig
-Dmapreduce.jobtracker.staging.root.dir=~/tmp/staging -x local
2012-10-19 11:06:49,624 [main] INFO  org.apache.pig.Main - Apache Pig
version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
2012-10-19 11:06:49,625 [main] INFO  org.apache.pig.Main - Logging error
messages to: /data1/home/ltang01/pig_1350670009621.log
2012-10-19 11:06:50,009 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: file:///
grunt> A = load 'toy.txt';
grunt> dump A;
2012-10-19 11:06:53,485 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2012-10-19 11:06:53,660 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2012-10-19 11:06:53,693 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2012-10-19 11:06:53,693 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2012-10-19 11:06:53,886 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2012-10-19 11:06:53,908 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2012-10-19 11:06:53,953 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2012-10-19 11:06:53,996 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
processName=JobTracker, sessionId2012-10-19 11:06:54,000 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2012-10-19 11:06:54,125 [Thread-5] WARN  org.apache.hadoop.mapred.JobClient
- No job jar file set.  User classes may not be found. See JobConf(Class)
or JobConf#setJar(String).
2012-10-19 11:06:54,207 [Thread-5] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2012-10-19 11:06:54,207 [Thread-5] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2012-10-19 11:06:54,222 [Thread-5] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2012-10-19 11:06:54,501 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2012-10-19 11:06:55,005 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_local_0001
2012-10-19 11:06:55,268 [Thread-6] INFO
org.apache.hadoop.mapreduce.util.ProcessTree - setsid exited with exit code
0
2012-10-19 11:06:55,824 [Thread-6] WARN
org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree -
/proc/<pid>/status does not have information about swap space used(VmSwap).
Can not track swap usage of a task.
2012-10-19 11:06:55,825 [Thread-6] INFO  org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@52cab854
2012-10-19 11:06:55,853 [Thread-6] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader
- Current split being processed file:/data1/home/ltang01/toy.txt:0+8
2012-10-19 11:06:55,914 [Thread-6] INFO  org.apache.hadoop.mapred.Task -
Task:attempt_local_0001_m_000000_0 is done. And is in the process of
commiting
2012-10-19 11:06:55,921 [Thread-6] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2012-10-19 11:06:55,922 [Thread-6] INFO  org.apache.hadoop.mapred.Task -
Task attempt_local_0001_m_000000_0 is allowed to commit now
2012-10-19 11:06:55,927 [Thread-6] INFO
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output
of task 'attempt_local_0001_m_000000_0' to
file:/tmp/temp487011820/tmp-1248641571
2012-10-19 11:06:55,927 [Thread-6] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2012-10-19 11:06:55,928 [Thread-6] INFO  org.apache.hadoop.mapred.Task -
Task 'attempt_local_0001_m_000000_0' done.
2012-10-19 11:06:55,929 [Thread-6] WARN
org.apache.hadoop.mapred.FileOutputCommitter - Output path is null in
cleanup
2012-10-19 11:06:56,380 [main] INFO  org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 0
time(s).
2012-10-19 11:06:57,382 [main] INFO  org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 1
time(s).
2012-10-19 11:06:58,383 [main] INFO  org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 2
time(s).
2012-10-19 11:06:59,385 [main] INFO  org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 3
time(s).
It keeps trying to connect to the local server. I have no idea what this is
about.  Can you help?

- Lei

On Wed, Oct 17, 2012 at 9:05 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB