Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Support for Hadoop 2.2


Copy link to this message
-
Re: Support for Hadoop 2.2
Hi Folks,

I am fairly new to Hadoop MR2 environment.  I have Hadoop0.23.1 (single
node) on Linux box and can successfully execute YARN test job, but when I
execute PIG (Ver. 12) to access and process files at HDFS, generate error
msg as follows.  I recompiled PIG with Hadoop23, but still no luck.

ant clean jar-withouthadoop -Dhadoopversion=23

Can someone please help and advise what I am missing here !!

Note: PIG in local mode works without any issue.

===============
*grunt> A = load 'NYSE_dividends' as (exch, symb, dt, div);*
2013-11-26 13:20:23,151 [main] WARN  org.apache.hadoop.conf.
Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:23,151 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:23,323 [main] WARN  org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:23,323 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
*grunt> dump A ;*
2013-11-26 13:20:31,805 [main] WARN  org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:31,806 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:31,841 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2013-11-26 13:20:31,913 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune,
DuplicateForEachColumnRewrite, GroupByConstParallelSetter,
ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter,
MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer,
PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter],
RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2013-11-26 13:20:32,159 [main] WARN  org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,160 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,266 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2013-11-26 13:20:32,304 [main] WARN  org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,304 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,318 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2013-11-26 13:20:32,319 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2013-11-26 13:20:32,322 [main] WARN  org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,322 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,498 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2013-11-26 13:20:32,504 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2013-11-26 13:20:32,504 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-11-26 13:20:32,504 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2013-11-26 13:20:32,504 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.output.compress is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress
2013-11-26 13:20:32,506 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- creating jar file Job1088179330840878220.jar
2013-11-26 13:20:35,473 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- jar file Job1088179330840878220.jar created
2013-11-26 13:20:35,473 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.jar is deprecated. Instead, use mapreduce.job.jar
2013-11-26 13:20:35,488 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2013-11-26 13:20:35,492 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Map only job, skipping reducer estimation
2013-11-26 13:20:35,513 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2013-11-26 13:20:35,514 [main] WARN  org.apache.hadoop.conf.Configuration -
mapred.job.tracker.http.address is deprecated. Instead, use
mapreduce.jobtracker.http.address
2013-11-26 13:20:35,954 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2013-11-26 13:20:35,954 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2013-11-26 13:20:35,971 [JobControl] WARN
org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2013-11-26 13:20:35,971 [JobControl] WARN
org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library not
loaded
2013-11-26 13:20:35,973 [JobControl] INFO
org.apache.hadoop.mapreduce.JobSubmitter - Cleaning up the staging area
null/hadoop/.sta
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB