Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Pig 0.3.0 and Hadoop 0.20.0


Copy link to this message
-
RE: Pig 0.3.0 and Hadoop 0.20.0
AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
parameters and thing should start working

-----Original Message-----
From: Naber, Chad [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, August 19, 2009 11:52 AM
To: [EMAIL PROTECTED]
Subject: RE: Pig 0.3.0 and Hadoop 0.20.0

If you are not using Hadoop on Demand (hod), you will need to comment
out the hod properties in pig's conf dir.

-----Original Message-----
From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, August 19, 2009 11:50 AM
To: [EMAIL PROTECTED]
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0

Getting closer...
Actually got and error this time:
ERROR 6012: Unable to run command: hod allocate ....

-Turner

On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
<[EMAIL PROTECTED]>wrote:

> You need a special hadoop20.jar to build the pig sources. Check out
PIG-660
>
> Santhosh
>
>
> -----Original Message-----
> From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> Sent: Wed 8/19/2009 9:53 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>
> Hm, still nothing.  Maybe I have to build it differently?  I will play
> around with the environment settings, but any more input is
appreciated.
>
> -Turner
>
> On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
<[EMAIL PROTECTED]
> >wrote:
>
> > Don't point it at mapred-site.xml directly, but the conf directory,
> > instead.
> > (it is smart enough to pick up the new config files when it's
running on
> > 20).
> >
> > I tried setting the path directly to mapred-site.xml like you have
it,
> > and indeed, that doesn't work. Changing to the directory should fix
> > your issue.
> >
> > -D
> >
> > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]>
> wrote:
> > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> working
> > > with 0.18.0 and had it working fine.
> > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
since
> > Pig
> > > is looking for hadoop-site.xml instead of mapred-site.xml.
> > > My environment looks like this now:
> > >
> > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > >
> >
>
PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
conf/mapred-site.xml
> > > PIGDIR=/usr/local/pig-0.3.0
> > > HADOOP_HOME=/usr/local/hadoop
> > > HADOOPDIR=/usr/local/hadoop/conf
> > > PIG_HADOOP_VERSION=20
> > >
> > > Pig still isn't connecting correctly.
> > >
> > > -Turner
> > >
> > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
<[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> Nipun and Turner,
> > >> What are you setting PIG_CLASSPATH to?
> > >>
> > >> My environment works if I set it to
> > >> /path/to/pig.jar:path/to/mapred-site.xml
> > >>
> > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> > >> breakage -- I haven't quite decided if that's a bug or not.)
> > >>
> > >> For completeness, a full set of environment variables that make
it
> > >> work for me. I am running the Cloudera CDH2 testing package for
> > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> version
> > >> of Pig from trunk.
> > >>
> > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> > >> PIG_HADOOP_VERSION=20
> > >> PIG_CONF_DIR=./conf/
> > >>
> > >>
> >
>
PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
/pig.jar
> > >>
> > >>
> > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
Kunkel<[EMAIL PROTECTED]>
> > wrote:
> > >> > Hello,
> > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig
no
> > longer
> > >> > connects to the HDFS.
> > >> > I tried rebuilding and applying patch PIG660.
> > >> > I have a script that I run that exports these environment
tables
> > >> correctly:
> > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > >> > Pig still connects to file:/// instead of HDFS.
> > >> > I rebuilt the Pig package so that there are no patches applied
at
> the
> > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
from
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB