Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Support for Hadoop 2.2


Copy link to this message
-
Re: Support for Hadoop 2.2
Uttam Kumar 2013-11-27, 00:54
Hi Folks,

I am fairly new to Hadoop MR2 environment.  I have Hadoop0.23.1 (single
node) on Linux box and can successfully execute YARN test job, but when I
execute PIG (Ver. 12) to access and process files at HDFS, generate error
msg as follows.  I recompiled PIG with Hadoop23, but still no luck.

ant clean jar-withouthadoop -Dhadoopversion=23

Can someone please help and advise what I am missing here !!

Note: PIG in local mode works without any issue.

===============
*grunt> A = load 'NYSE_dividends' as (exch, symb, dt, div);*
2013-11-26 13:20:23,151 [main] WARN  org.apache.hadoop.conf.
Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:23,151 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:23,323 [main] WARN  org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:23,323 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
*grunt> dump A ;*
2013-11-26 13:20:31,805 [main] WARN  org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:31,806 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:31,841 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2013-11-26 13:20:31,913 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune,
DuplicateForEachColumnRewrite, GroupByConstParallelSetter,
ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter,
MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer,
PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter],
RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2013-11-26 13:20:32,159 [main] WARN  org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,160 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,266 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2013-11-26 13:20:32,304 [main] WARN  org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,304 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,318 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2013-11-26 13:20:32,319 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2013-11-26 13:20:32,322 [main] WARN  org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,322 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,498 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2013-11-26 13:20:32,504 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2013-11-26 13:20:32,504 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-11-26 13:20:32,504 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2013-11-26 13:20:32,504 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.output.compress is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress
2013-11-26 13:20:32,506 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- creating jar file Job1088179330840878220.jar
2013-11-26 13:20:35,473 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- jar file Job1088179330840878220.jar created
2013-11-26 13:20:35,473 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.jar is deprecated. Instead, use mapreduce.job.jar
2013-11-26 13:20:35,488 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2013-11-26 13:20:35,492 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Map only job, skipping reducer estimation
2013-11-26 13:20:35,513 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2013-11-26 13:20:35,514 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker.http.address is deprecated. Instead, use
mapreduce.jobtracker.http.address
2013-11-26 13:20:35,954 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2013-11-26 13:20:35,954 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2013-11-26 13:20:35,971 [JobControl] WARN
org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2013-11-26 13:20:35,971 [JobControl] WARN
org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library not
loaded
2013-11-26 13:20:35,973 [JobControl] INFO
org.apache.hadoop.mapreduce.JobSubmitter - Cleaning up the staging area
null/hadoop/.sta