Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Help on running pig in local mode


Copy link to this message
-
Re: Help on running pig in local mode
What are you running this on?

This is really odd:
/proc/<pid>/status does not have information about swap space used(VmSwap).

D

On Fri, Oct 19, 2012 at 11:11 AM, lei tang <[EMAIL PROTECTED]> wrote:
> I checked the hadoop config and found that the staging dir specified does
> not exist in local path (only in hdfs) and neither do I have write access.
> So I changed the staging dir as below, but encounter some weird error
> message:
>
> [ltang01@stg-trgt00 ~]$ pig
> -Dmapreduce.jobtracker.staging.root.dir=~/tmp/staging -x local
> 2012-10-19 11:06:49,624 [main] INFO  org.apache.pig.Main - Apache Pig
> version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
> 2012-10-19 11:06:49,625 [main] INFO  org.apache.pig.Main - Logging error
> messages to: /data1/home/ltang01/pig_1350670009621.log
> 2012-10-19 11:06:50,009 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: file:///
> grunt> A = load 'toy.txt';
> grunt> dump A;
> 2012-10-19 11:06:53,485 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2012-10-19 11:06:53,660 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2012-10-19 11:06:53,693 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2012-10-19 11:06:53,693 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2012-10-19 11:06:53,886 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to the job
> 2012-10-19 11:06:53,908 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2012-10-19 11:06:53,953 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2012-10-19 11:06:53,996 [main] INFO
> org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
> processName=JobTracker, sessionId> 2012-10-19 11:06:54,000 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2012-10-19 11:06:54,125 [Thread-5] WARN  org.apache.hadoop.mapred.JobClient
> - No job jar file set.  User classes may not be found. See JobConf(Class)
> or JobConf#setJar(String).
> 2012-10-19 11:06:54,207 [Thread-5] INFO
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
> to process : 1
> 2012-10-19 11:06:54,207 [Thread-5] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths to process : 1
> 2012-10-19 11:06:54,222 [Thread-5] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths (combined) to process : 1
> 2012-10-19 11:06:54,501 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2012-10-19 11:06:55,005 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - HadoopJobId: job_local_0001
> 2012-10-19 11:06:55,268 [Thread-6] INFO
> org.apache.hadoop.mapreduce.util.ProcessTree - setsid exited with exit code
> 0
> 2012-10-19 11:06:55,824 [Thread-6] WARN
> org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree -
> /proc/<pid>/status does not have information about swap space used(VmSwap).
> Can not track swap usage of a task.
> 2012-10-19 11:06:55,825 [Thread-6] INFO  org.apache.hadoop.mapred.Task -
> Using ResourceCalculatorPlugin :
> org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@52cab854
> 2012-10-19 11:06:55,853 [Thread-6] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader
> - Current split being processed file:/data1/home/ltang01/toy.txt:0+8