Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Pig 0.3.0 and Hadoop 0.20.0


Copy link to this message
-
RE: Pig 0.3.0 and Hadoop 0.20.0
You need a special hadoop20.jar to build the pig sources. Check out PIG-660

Santhosh
-----Original Message-----
From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
Sent: Wed 8/19/2009 9:53 AM
To: [EMAIL PROTECTED]
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
 
Hm, still nothing.  Maybe I have to build it differently?  I will play
around with the environment settings, but any more input is appreciated.

-Turner

On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]>wrote:

> Don't point it at mapred-site.xml directly, but the conf directory,
> instead.
> (it is smart enough to pick up the new config files when it's running on
> 20).
>
> I tried setting the path directly to mapred-site.xml like you have it,
> and indeed, that doesn't work. Changing to the directory should fix
> your issue.
>
> -D
>
> On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> > I am running off of unpacked tar.gz files of Hadoop and Pig. Was working
> > with 0.18.0 and had it working fine.
> > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since
> Pig
> > is looking for hadoop-site.xml instead of mapred-site.xml.
> > My environment looks like this now:
> >
> > JAVA_HOME=/usr/lib/jvm/java-6-sun
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> > PIGDIR=/usr/local/pig-0.3.0
> > HADOOP_HOME=/usr/local/hadoop
> > HADOOPDIR=/usr/local/hadoop/conf
> > PIG_HADOOP_VERSION=20
> >
> > Pig still isn't connecting correctly.
> >
> > -Turner
> >
> > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]
> >wrote:
> >
> >> Nipun and Turner,
> >> What are you setting PIG_CLASSPATH to?
> >>
> >> My environment works if I set it to
> >> /path/to/pig.jar:path/to/mapred-site.xml
> >>
> >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> >> breakage -- I haven't quite decided if that's a bug or not.)
> >>
> >> For completeness, a full set of environment variables that make it
> >> work for me. I am running the Cloudera CDH2 testing package for
> >> debian/ubuntu (from http://archive.cloudera.com) and a patched version
> >> of Pig from trunk.
> >>
> >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> >> PIG_HADOOP_VERSION=20
> >> PIG_CONF_DIR=./conf/
> >>
> >>
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
> >>
> >>
> >> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<[EMAIL PROTECTED]>
> wrote:
> >> > Hello,
> >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no
> longer
> >> > connects to the HDFS.
> >> > I tried rebuilding and applying patch PIG660.
> >> > I have a script that I run that exports these environment tables
> >> correctly:
> >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> >> > Pig still connects to file:/// instead of HDFS.
> >> > I rebuilt the Pig package so that there are no patches applied at the
> >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> >> >
> >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the
> >> start?
> >> >
> >> > Thanks!
> >> >
> >> > --
> >> >
> >> > -Turner Kunkel
> >> >
> >>
> >
>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB