Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Unable to run PIG local as regular user (non-root)

Watrous, Daniel 2013-11-27, 19:01
Cheolsoo Park 2013-12-02, 04:54
Copy link to this message
RE: Unable to run PIG local as regular user (non-root)

I hadn't noticed that. I do have hadoop 2.2.0 installed. Does the PIG distribution include hadoop v1? I suppose that if my environment variables aren't set correctly for hadoop 2 as root, then it may default to version 1 of hadoop. The reason for this is that the path to hadoop/bin is part of PATH as my user, but not as root.

I have confirmed this by removing the path to hadoop binaries from my PATH environment variable and running it locally. Everything worked identically. To correct this permanently I should either move backward to version 1.x of hadoop or compile PIG with hadoop 2.x support. Has PIG been tested against hadoop 2.x?


-----Original Message-----
From: Cheolsoo Park [mailto:[EMAIL PROTECTED]]
Sent: Sunday, December 01, 2013 9:55 PM
Subject: Re: Unable to run PIG local as regular user (non-root)

Hi Daniel,

>From the console output, I can see Pig ran against Hadoop 1.x as root while it ran against Hadoop 2.x as watrous. I don't know what your setup is like, but please double check what version of Hadoop jars are present in classpath. Running "pig -secretDebugCmd" should show HADOOP_CLASSPATH.

Apparently, your Pig jar is compiled against Hadoop 1.x, so it only works against Hadoop 1.x (as root).


On Wed, Nov 27, 2013 at 11:01 AM, Watrous, Daniel

> Hi,
> I've been following the book Hadoop Definitive Guide and a few other
> resources while exploring PIG. I am able to run as root, but not as a
> regular user. I am running in local mode. I've found a few links, but
> they haven't helped me solve my issue. The most relevant seems to be:
> http://answers.mapr.com/questions/5135/cannot-run-pig-in-local-mode
> My first experiment is just to load in a text file and dump it to the
> console. The files exist and are owned by the user running PIG. I show
> the failed attempt as a regular user followed by a successful run as
> root on the same system. Any help is appreciated.
> Here's what I'm trying:
> [watrous@c0003913 ~]$ pig -x local
> 2013-11-27 18:53:18,691 [main] INFO  org.apache.pig.Main - Apache Pig
> version 0.12.0 (r1529718) compiled Oct 07 2013, 12:20:14
> 2013-11-27 18:53:18,692 [main] INFO  org.apache.pig.Main - Logging
> error messages to: /home/watrous/pig_1385578398689.log
> 2013-11-27 18:53:18,747 [main] INFO  org.apache.pig.impl.util.Utils -
> Default bootup file /home/watrous/.pigbootup not found
> 2013-11-27 18:53:19,025 [main] INFO
>  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
> deprecated. Instead, use fs.defaultFS
> 2013-11-27 18:53:19,025 [main] INFO
>  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker
> is deprecated. Instead, use mapreduce.jobtracker.address
> 2013-11-27 18:53:19,027 [main] INFO
>  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> Connecting to hadoop file system at: file:///
> 2013-11-27 18:53:19,031 [main] INFO
>  org.apache.hadoop.conf.Configuration.deprecation -
> mapred.used.genericoptionsparser is deprecated. Instead, use
> mapreduce.client.genericoptionsparser.used
> 2013-11-27 18:53:19,451 [main] WARN
>  org.apache.hadoop.util.NativeCodeLoader - Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable
> 2013-11-27 18:53:19,517 [main] INFO
>  org.apache.hadoop.conf.Configuration.deprecation -
> io.bytes.per.checksum is deprecated. Instead, use
> dfs.bytes-per-checksum
> 2013-11-27 18:53:19,519 [main] INFO
>  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
> deprecated. Instead, use fs.defaultFS
> grunt> ls /home/watrous/input/
> file:/home/watrous/input/sample.txt<r 1>        530
> grunt> cat /home/watrous/input/sample.txt
> 0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999
> 0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999
Cheolsoo Park 2013-12-02, 18:34