Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Pig 0.3.0 and Hadoop 0.20.0


Copy link to this message
-
RE: Pig 0.3.0 and Hadoop 0.20.0
You need a special hadoop20.jar to build the pig sources. Check out PIG-660

Santhosh
-----Original Message-----
From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
Sent: Wed 8/19/2009 9:53 AM
To: [EMAIL PROTECTED]
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
 
Hm, still nothing.  Maybe I have to build it differently?  I will play
around with the environment settings, but any more input is appreciated.

-Turner

On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]>wrote:

> Don't point it at mapred-site.xml directly, but the conf directory,
> instead.
> (it is smart enough to pick up the new config files when it's running on
> 20).
>
> I tried setting the path directly to mapred-site.xml like you have it,
> and indeed, that doesn't work. Changing to the directory should fix
> your issue.
>
> -D
>
> On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> > I am running off of unpacked tar.gz files of Hadoop and Pig. Was working
> > with 0.18.0 and had it working fine.
> > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since
> Pig
> > is looking for hadoop-site.xml instead of mapred-site.xml.
> > My environment looks like this now:
> >
> > JAVA_HOME=/usr/lib/jvm/java-6-sun
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> > PIGDIR=/usr/local/pig-0.3.0
> > HADOOP_HOME=/usr/local/hadoop
> > HADOOPDIR=/usr/local/hadoop/conf
> > PIG_HADOOP_VERSION=20
> >
> > Pig still isn't connecting correctly.
> >
> > -Turner
> >
> > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]
> >wrote:
> >
> >> Nipun and Turner,
> >> What are you setting PIG_CLASSPATH to?
> >>
> >> My environment works if I set it to
> >> /path/to/pig.jar:path/to/mapred-site.xml
> >>
> >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> >> breakage -- I haven't quite decided if that's a bug or not.)
> >>
> >> For completeness, a full set of environment variables that make it
> >> work for me. I am running the Cloudera CDH2 testing package for
> >> debian/ubuntu (from http://archive.cloudera.com) and a patched version
> >> of Pig from trunk.
> >>
> >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> >> PIG_HADOOP_VERSION=20
> >> PIG_CONF_DIR=./conf/
> >>
> >>
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
> >>
> >>
> >> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<[EMAIL PROTECTED]>
> wrote:
> >> > Hello,
> >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no
> longer
> >> > connects to the HDFS.
> >> > I tried rebuilding and applying patch PIG660.
> >> > I have a script that I run that exports these environment tables
> >> correctly:
> >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> >> > Pig still connects to file:/// instead of HDFS.
> >> > I rebuilt the Pig package so that there are no patches applied at the
> >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> >> >
> >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the
> >> start?
> >> >
> >> > Thanks!
> >> >
> >> > --
> >> >
> >> > -Turner Kunkel
> >> >
> >>
> >
>