Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Pig 0.3.0 and Hadoop 0.20.0


Copy link to this message
-
RE: Pig 0.3.0 and Hadoop 0.20.0
AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
parameters and thing should start working

-----Original Message-----
From: Naber, Chad [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, August 19, 2009 11:52 AM
To: [EMAIL PROTECTED]
Subject: RE: Pig 0.3.0 and Hadoop 0.20.0

If you are not using Hadoop on Demand (hod), you will need to comment
out the hod properties in pig's conf dir.

-----Original Message-----
From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, August 19, 2009 11:50 AM
To: [EMAIL PROTECTED]
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0

Getting closer...
Actually got and error this time:
ERROR 6012: Unable to run command: hod allocate ....

-Turner

On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
<[EMAIL PROTECTED]>wrote:

> You need a special hadoop20.jar to build the pig sources. Check out
PIG-660
>
> Santhosh
>
>
> -----Original Message-----
> From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> Sent: Wed 8/19/2009 9:53 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>
> Hm, still nothing.  Maybe I have to build it differently?  I will play
> around with the environment settings, but any more input is
appreciated.
>
> -Turner
>
> On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
<[EMAIL PROTECTED]
> >wrote:
>
> > Don't point it at mapred-site.xml directly, but the conf directory,
> > instead.
> > (it is smart enough to pick up the new config files when it's
running on
> > 20).
> >
> > I tried setting the path directly to mapred-site.xml like you have
it,
> > and indeed, that doesn't work. Changing to the directory should fix
> > your issue.
> >
> > -D
> >
> > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]>
> wrote:
> > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> working
> > > with 0.18.0 and had it working fine.
> > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
since
> > Pig
> > > is looking for hadoop-site.xml instead of mapred-site.xml.
> > > My environment looks like this now:
> > >
> > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > >
> >
>
PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
conf/mapred-site.xml
> > > PIGDIR=/usr/local/pig-0.3.0
> > > HADOOP_HOME=/usr/local/hadoop
> > > HADOOPDIR=/usr/local/hadoop/conf
> > > PIG_HADOOP_VERSION=20
> > >
> > > Pig still isn't connecting correctly.
> > >
> > > -Turner
> > >
> > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
<[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> Nipun and Turner,
> > >> What are you setting PIG_CLASSPATH to?
> > >>
> > >> My environment works if I set it to
> > >> /path/to/pig.jar:path/to/mapred-site.xml
> > >>
> > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> > >> breakage -- I haven't quite decided if that's a bug or not.)
> > >>
> > >> For completeness, a full set of environment variables that make
it
> > >> work for me. I am running the Cloudera CDH2 testing package for
> > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> version
> > >> of Pig from trunk.
> > >>
> > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> > >> PIG_HADOOP_VERSION=20
> > >> PIG_CONF_DIR=./conf/
> > >>
> > >>
> >
>
PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
/pig.jar
> > >>
> > >>
> > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
Kunkel<[EMAIL PROTECTED]>
> > wrote:
> > >> > Hello,
> > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig
no
> > longer
> > >> > connects to the HDFS.
> > >> > I tried rebuilding and applying patch PIG660.
> > >> > I have a script that I run that exports these environment
tables
> > >> correctly:
> > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > >> > Pig still connects to file:/// instead of HDFS.
> > >> > I rebuilt the Pig package so that there are no patches applied
at
> the
> > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
from