Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Help on running pig in local mode


+
lei tang 2012-10-15, 22:43
+
Prashant Kommireddi 2012-10-15, 22:45
+
lei tang 2012-10-15, 23:35
+
Dmitriy Ryaboy 2012-10-18, 04:05
+
lei tang 2012-10-19, 18:11
Copy link to this message
-
Re: Help on running pig in local mode
What are you running this on?

This is really odd:
/proc/<pid>/status does not have information about swap space used(VmSwap).

D

On Fri, Oct 19, 2012 at 11:11 AM, lei tang <[EMAIL PROTECTED]> wrote:
> I checked the hadoop config and found that the staging dir specified does
> not exist in local path (only in hdfs) and neither do I have write access.
> So I changed the staging dir as below, but encounter some weird error
> message:
>
> [ltang01@stg-trgt00 ~]$ pig
> -Dmapreduce.jobtracker.staging.root.dir=~/tmp/staging -x local
> 2012-10-19 11:06:49,624 [main] INFO  org.apache.pig.Main - Apache Pig
> version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
> 2012-10-19 11:06:49,625 [main] INFO  org.apache.pig.Main - Logging error
> messages to: /data1/home/ltang01/pig_1350670009621.log
> 2012-10-19 11:06:50,009 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: file:///
> grunt> A = load 'toy.txt';
> grunt> dump A;
> 2012-10-19 11:06:53,485 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2012-10-19 11:06:53,660 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2012-10-19 11:06:53,693 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2012-10-19 11:06:53,693 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2012-10-19 11:06:53,886 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to the job
> 2012-10-19 11:06:53,908 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2012-10-19 11:06:53,953 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2012-10-19 11:06:53,996 [main] INFO
> org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
> processName=JobTracker, sessionId> 2012-10-19 11:06:54,000 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2012-10-19 11:06:54,125 [Thread-5] WARN  org.apache.hadoop.mapred.JobClient
> - No job jar file set.  User classes may not be found. See JobConf(Class)
> or JobConf#setJar(String).
> 2012-10-19 11:06:54,207 [Thread-5] INFO
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
> to process : 1
> 2012-10-19 11:06:54,207 [Thread-5] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths to process : 1
> 2012-10-19 11:06:54,222 [Thread-5] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths (combined) to process : 1
> 2012-10-19 11:06:54,501 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2012-10-19 11:06:55,005 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - HadoopJobId: job_local_0001
> 2012-10-19 11:06:55,268 [Thread-6] INFO
> org.apache.hadoop.mapreduce.util.ProcessTree - setsid exited with exit code
> 0
> 2012-10-19 11:06:55,824 [Thread-6] WARN
> org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree -
> /proc/<pid>/status does not have information about swap space used(VmSwap).
> Can not track swap usage of a task.
> 2012-10-19 11:06:55,825 [Thread-6] INFO  org.apache.hadoop.mapred.Task -
> Using ResourceCalculatorPlugin :
> org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@52cab854
> 2012-10-19 11:06:55,853 [Thread-6] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader
> - Current split being processed file:/data1/home/ltang01/toy.txt:0+8
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB