Pig, mail # user - RE: Pig 0.3.0 and Hadoop 0.20.0

Re: Pig 0.3.0 and Hadoop 0.20.0

Dmitriy Ryaboy 2009-08-18, 21:59

Nipun and Turner,
What are you setting PIG_CLASSPATH to?

My environment works if I set it to /path/to/pig.jar:path/to/mapred-site.xml

(leaving off the path to mapred-site.xml or pig.jar both lead to
breakage -- I haven't quite decided if that's a bug or not.)

For completeness, a full set of environment variables that make it
work for me. I am running the Cloudera CDH2 testing package for
debian/ubuntu (from http://archive.cloudera.com) and a patched version
of Pig from trunk.

JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
PIGDIR=/home/dvryaboy/src/pig/trunk/
PIG_HADOOP_VERSION=20
PIG_CONF_DIR=./conf/
PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> Hello,
> I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no longer
> connects to the HDFS.
> I tried rebuilding and applying patch PIG660.
> I have a script that I run that exports these environment tables correctly:
> JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> Pig still connects to file:/// instead of HDFS.
> I rebuilt the Pig package so that there are no patches applied at the
> moment, as if I had just unpacked PIG-0.3.0 for the first time.
>
> So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the start?
>
> Thanks!
>
> --
>
> -Turner Kunkel
>

Re: Pig 0.3.0 and Hadoop 0.20.0

Turner Kunkel 2009-08-19, 13:26

I am running off of unpacked tar.gz files of Hadoop and Pig. Was working
with 0.18.0 and had it working fine.
With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since Pig
is looking for hadoop-site.xml instead of mapred-site.xml.
My environment looks like this now:

JAVA_HOME=/usr/lib/jvm/java-6-sun
PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
PIGDIR=/usr/local/pig-0.3.0
HADOOP_HOME=/usr/local/hadoop
HADOOPDIR=/usr/local/hadoop/conf
PIG_HADOOP_VERSION=20

Pig still isn't connecting correctly.

-Turner

On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]>wrote:

> Nipun and Turner,
> What are you setting PIG_CLASSPATH to?
>
> My environment works if I set it to
> /path/to/pig.jar:path/to/mapred-site.xml
>
> (leaving off the path to mapred-site.xml or pig.jar both lead to
> breakage -- I haven't quite decided if that's a bug or not.)
>
> For completeness, a full set of environment variables that make it
> work for me. I am running the Cloudera CDH2 testing package for
> debian/ubuntu (from http://archive.cloudera.com) and a patched version
> of Pig from trunk.
>
> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> PIGDIR=/home/dvryaboy/src/pig/trunk/
> PIG_HADOOP_VERSION=20
> PIG_CONF_DIR=./conf/
>
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
>
>
> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> > Hello,
> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no longer
> > connects to the HDFS.
> > I tried rebuilding and applying patch PIG660.
> > I have a script that I run that exports these environment tables
> correctly:
> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > Pig still connects to file:/// instead of HDFS.
> > I rebuilt the Pig package so that there are no patches applied at the
> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> >
> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the
> start?
> >
> > Thanks!
> >
> > --
> >
> > -Turner Kunkel
> >
>

Re: Pig 0.3.0 and Hadoop 0.20.0

Dmitriy Ryaboy 2009-08-19, 15:09

Don't point it at mapred-site.xml directly, but the conf directory, instead.
(it is smart enough to pick up the new config files when it's running on 20).

I tried setting the path directly to mapred-site.xml like you have it,
and indeed, that doesn't work. Changing to the directory should fix
your issue.

-D

On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> I am running off of unpacked tar.gz files of Hadoop and Pig. Was working
> with 0.18.0 and had it working fine.
> With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since Pig
> is looking for hadoop-site.xml instead of mapred-site.xml.
> My environment looks like this now:
>
> JAVA_HOME=/usr/lib/jvm/java-6-sun
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> PIGDIR=/usr/local/pig-0.3.0
> HADOOP_HOME=/usr/local/hadoop
> HADOOPDIR=/usr/local/hadoop/conf
> PIG_HADOOP_VERSION=20
>
> Pig still isn't connecting correctly.
>
> -Turner
>
> On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]>wrote:
>
>> Nipun and Turner,
>> What are you setting PIG_CLASSPATH to?
>>
>> My environment works if I set it to
>> /path/to/pig.jar:path/to/mapred-site.xml
>>
>> (leaving off the path to mapred-site.xml or pig.jar both lead to
>> breakage -- I haven't quite decided if that's a bug or not.)
>>
>> For completeness, a full set of environment variables that make it
>> work for me. I am running the Cloudera CDH2 testing package for
>> debian/ubuntu (from http://archive.cloudera.com) and a patched version
>> of Pig from trunk.
>>
>> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
>> PIGDIR=/home/dvryaboy/src/pig/trunk/
>> PIG_HADOOP_VERSION=20
>> PIG_CONF_DIR=./conf/
>>
>> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
>>
>>
>> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
>> > Hello,
>> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no longer
>> > connects to the HDFS.
>> > I tried rebuilding and applying patch PIG660.
>> > I have a script that I run that exports these environment tables
>> correctly:
>> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
>> > Pig still connects to file:/// instead of HDFS.
>> > I rebuilt the Pig package so that there are no patches applied at the
>> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
>> >
>> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the
>> start?
>> >
>> > Thanks!
>> >
>> > --
>> >
>> > -Turner Kunkel
>> >
>>
>

Re: Pig 0.3.0 and Hadoop 0.20.0

Turner Kunkel 2009-08-19, 16:53

Hm, still nothing. Maybe I have to build it differently? I will play
around with the environment settings, but any more input is appreciated.

-Turner

On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]>wrote:

> Don't point it at mapred-site.xml directly, but the conf directory,
> instead.
> (it is smart enough to pick up the new config files when it's running on
> 20).
>
> I tried setting the path directly to mapred-site.xml like you have it,
> and indeed, that doesn't work. Changing to the directory should fix
> your issue.
>
> -D
>
> On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> > I am running off of unpacked tar.gz files of Hadoop and Pig. Was working
> > with 0.18.0 and had it working fine.
> > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since
> Pig
> > is looking for hadoop-site.xml instead of mapred-site.xml.
> > My environment looks like this now:
> >
> > JAVA_HOME=/usr/lib/jvm/java-6-sun
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> > PIGDIR=/usr/local/pig-0.3.0
> > HADOOP_HOME=/usr/local/hadoop
> > HADOOPDIR=/usr/local/hadoop/conf
> > PIG_HADOOP_VERSION=20
> >
> > Pig still isn't connecting correctly.
> >
> > -Turner
> >
> > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]
> >wrote:
> >
> >> Nipun and Turner,
> >> What are you setting PIG_CLASSPATH to?
> >>
> >> My environment works if I set it to
> >> /path/to/pig.jar:path/to/mapred-site.xml
> >>
> >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> >> breakage -- I haven't quite decided if that's a bug or not.)
> >>
> >> For completeness, a full set of environment variables that make it
> >> work for me. I am running the Cloudera CDH2 testing package for
> >> debian/ubuntu (from http://archive.cloudera.com) and a patched version
> >> of Pig from trunk.
> >>
> >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> >> PIG_HADOOP_VERSION=20
> >> PIG_CONF_DIR=./conf/
> >>
> >>
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
> >>
> >>
> >> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<[EMAIL PROTECTED]>
> wrote:
> >> > Hello,
> >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no
> longer
> >> > connects to the HDFS.
> >> > I tried rebuilding and applying patch PIG660.
> >> > I have a script that I run that exports these environment tables
> >> correctly:
> >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> >> > Pig still connects to file:/// instead of HDFS.
> >> > I rebuilt the Pig package so that there are no patches applied at the
> >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> >> >
> >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the
> >> start?
> >> >
> >> > Thanks!
> >> >
> >> > --
> >> >
> >> > -Turner Kunkel
> >> >
> >>
> >
>

RE: Pig 0.3.0 and Hadoop 0.20.0

Santhosh Srinivasan 2009-08-19, 17:24

You need a special hadoop20.jar to build the pig sources. Check out PIG-660

Santhosh
-----Original Message-----
From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
Sent: Wed 8/19/2009 9:53 AM
To: [EMAIL PROTECTED]
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0

Hm, still nothing. Maybe I have to build it differently? I will play
around with the environment settings, but any more input is appreciated.

-Turner

On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]>wrote:

> Don't point it at mapred-site.xml directly, but the conf directory,
> instead.
> (it is smart enough to pick up the new config files when it's running on
> 20).
>
> I tried setting the path directly to mapred-site.xml like you have it,
> and indeed, that doesn't work. Changing to the directory should fix
> your issue.
>
> -D
>
> On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> > I am running off of unpacked tar.gz files of Hadoop and Pig. Was working
> > with 0.18.0 and had it working fine.
> > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since
> Pig
> > is looking for hadoop-site.xml instead of mapred-site.xml.
> > My environment looks like this now:
> >
> > JAVA_HOME=/usr/lib/jvm/java-6-sun
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> > PIGDIR=/usr/local/pig-0.3.0
> > HADOOP_HOME=/usr/local/hadoop
> > HADOOPDIR=/usr/local/hadoop/conf
> > PIG_HADOOP_VERSION=20
> >
> > Pig still isn't connecting correctly.
> >
> > -Turner
> >
> > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]
> >wrote:
> >
> >> Nipun and Turner,
> >> What are you setting PIG_CLASSPATH to?
> >>
> >> My environment works if I set it to
> >> /path/to/pig.jar:path/to/mapred-site.xml
> >>
> >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> >> breakage -- I haven't quite decided if that's a bug or not.)
> >>
> >> For completeness, a full set of environment variables that make it
> >> work for me. I am running the Cloudera CDH2 testing package for
> >> debian/ubuntu (from http://archive.cloudera.com) and a patched version
> >> of Pig from trunk.
> >>
> >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> >> PIG_HADOOP_VERSION=20
> >> PIG_CONF_DIR=./conf/
> >>
> >>
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
> >>
> >>
> >> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<[EMAIL PROTECTED]>
> wrote:
> >> > Hello,
> >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no
> longer
> >> > connects to the HDFS.
> >> > I tried rebuilding and applying patch PIG660.
> >> > I have a script that I run that exports these environment tables
> >> correctly:
> >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> >> > Pig still connects to file:/// instead of HDFS.
> >> > I rebuilt the Pig package so that there are no patches applied at the
> >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> >> >
> >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the
> >> start?
> >> >
> >> > Thanks!
> >> >
> >> > --
> >> >
> >> > -Turner Kunkel
> >> >
> >>
> >
>

Re: Pig 0.3.0 and Hadoop 0.20.0

Turner Kunkel 2009-08-19, 18:50

Getting closer...
Actually got and error this time:
ERROR 6012: Unable to run command: hod allocate ....

-Turner

On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan <[EMAIL PROTECTED]>wrote:

> You need a special hadoop20.jar to build the pig sources. Check out PIG-660
>
> Santhosh
>
>
> -----Original Message-----
> From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> Sent: Wed 8/19/2009 9:53 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>
> Hm, still nothing. Maybe I have to build it differently? I will play
> around with the environment settings, but any more input is appreciated.
>
> -Turner
>
> On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]
> >wrote:
>
> > Don't point it at mapred-site.xml directly, but the conf directory,
> > instead.
> > (it is smart enough to pick up the new config files when it's running on
> > 20).
> >
> > I tried setting the path directly to mapred-site.xml like you have it,
> > and indeed, that doesn't work. Changing to the directory should fix
> > your issue.
> >
> > -D
> >
> > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]>
> wrote:
> > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> working
> > > with 0.18.0 and had it working fine.
> > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since
> > Pig
> > > is looking for hadoop-site.xml instead of mapred-site.xml.
> > > My environment looks like this now:
> > >
> > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > >
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> > > PIGDIR=/usr/local/pig-0.3.0
> > > HADOOP_HOME=/usr/local/hadoop
> > > HADOOPDIR=/usr/local/hadoop/conf
> > > PIG_HADOOP_VERSION=20
> > >
> > > Pig still isn't connecting correctly.
> > >
> > > -Turner
> > >
> > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> Nipun and Turner,
> > >> What are you setting PIG_CLASSPATH to?
> > >>
> > >> My environment works if I set it to
> > >> /path/to/pig.jar:path/to/mapred-site.xml
> > >>
> > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> > >> breakage -- I haven't quite decided if that's a bug or not.)
> > >>
> > >> For completeness, a full set of environment variables that make it
> > >> work for me. I am running the Cloudera CDH2 testing package for
> > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> version
> > >> of Pig from trunk.
> > >>
> > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> > >> PIG_HADOOP_VERSION=20
> > >> PIG_CONF_DIR=./conf/
> > >>
> > >>
> >
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
> > >>
> > >>
> > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<[EMAIL PROTECTED]>
> > wrote:
> > >> > Hello,
> > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no
> > longer
> > >> > connects to the HDFS.
> > >> > I tried rebuilding and applying patch PIG660.
> > >> > I have a script that I run that exports these environment tables
> > >> correctly:
> > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > >> > Pig still connects to file:/// instead of HDFS.
> > >> > I rebuilt the Pig package so that there are no patches applied at
> the
> > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> > >> >
> > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from
> the
> > >> start?
> > >> >
> > >> > Thanks!
> > >> >
> > >> > --
> > >> >
> > >> > -Turner Kunkel
> > >> >
> > >>
> > >
> >
>

RE: Pig 0.3.0 and Hadoop 0.20.0

Naber, Chad 2009-08-19, 18:52

If you are not using Hadoop on Demand (hod), you will need to comment out the hod properties in pig's conf dir.

-----Original Message-----
From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, August 19, 2009 11:50 AM
To: [EMAIL PROTECTED]
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0

Getting closer...
Actually got and error this time:
ERROR 6012: Unable to run command: hod allocate ....

-Turner

On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan <[EMAIL PROTECTED]>wrote:

> You need a special hadoop20.jar to build the pig sources. Check out PIG-660
>
> Santhosh
>
>
> -----Original Message-----
> From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> Sent: Wed 8/19/2009 9:53 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>
> Hm, still nothing. Maybe I have to build it differently? I will play
> around with the environment settings, but any more input is appreciated.
>
> -Turner
>
> On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]
> >wrote:
>
> > Don't point it at mapred-site.xml directly, but the conf directory,
> > instead.
> > (it is smart enough to pick up the new config files when it's running on
> > 20).
> >
> > I tried setting the path directly to mapred-site.xml like you have it,
> > and indeed, that doesn't work. Changing to the directory should fix
> > your issue.
> >
> > -D
> >
> > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]>
> wrote:
> > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> working
> > > with 0.18.0 and had it working fine.
> > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since
> > Pig
> > > is looking for hadoop-site.xml instead of mapred-site.xml.
> > > My environment looks like this now:
> > >
> > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > >
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> > > PIGDIR=/usr/local/pig-0.3.0
> > > HADOOP_HOME=/usr/local/hadoop
> > > HADOOPDIR=/usr/local/hadoop/conf
> > > PIG_HADOOP_VERSION=20
> > >
> > > Pig still isn't connecting correctly.
> > >
> > > -Turner
> > >
> > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> Nipun and Turner,
> > >> What are you setting PIG_CLASSPATH to?
> > >>
> > >> My environment works if I set it to
> > >> /path/to/pig.jar:path/to/mapred-site.xml
> > >>
> > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> > >> breakage -- I haven't quite decided if that's a bug or not.)
> > >>
> > >> For completeness, a full set of environment variables that make it
> > >> work for me. I am running the Cloudera CDH2 testing package for
> > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> version
> > >> of Pig from trunk.
> > >>
> > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> > >> PIG_HADOOP_VERSION=20
> > >> PIG_CONF_DIR=./conf/
> > >>
> > >>
> >
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
> > >>
> > >>
> > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<[EMAIL PROTECTED]>
> > wrote:
> > >> > Hello,
> > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no
> > longer
> > >> > connects to the HDFS.
> > >> > I tried rebuilding and applying patch PIG660.
> > >> > I have a script that I run that exports these environment tables
> > >> correctly:
> > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > >> > Pig still connects to file:/// instead of HDFS.
> > >> > I rebuilt the Pig package so that there are no patches applied at
> the
> > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> > >> >
> > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from
> the
> > >> start?
> > >> >
> > >> > Thanks!
> > >> >
> > >> > --
> > >> >
> > >> > -Turner Kunkel
> > >> >
> > >>
> > >
> >
>

RE: Pig 0.3.0 and Hadoop 0.20.0

Santhosh Srinivasan 2009-08-19, 19:06

AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
parameters and thing should start working

-----Original Message-----
From: Naber, Chad [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, August 19, 2009 11:52 AM
To: [EMAIL PROTECTED]
Subject: RE: Pig 0.3.0 and Hadoop 0.20.0

If you are not using Hadoop on Demand (hod), you will need to comment
out the hod properties in pig's conf dir.

-----Original Message-----
From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, August 19, 2009 11:50 AM
To: [EMAIL PROTECTED]
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0

Getting closer...
Actually got and error this time:
ERROR 6012: Unable to run command: hod allocate ....

-Turner

On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
<[EMAIL PROTECTED]>wrote:

> You need a special hadoop20.jar to build the pig sources. Check out
PIG-660
>
> Santhosh
>
>
> -----Original Message-----
> From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> Sent: Wed 8/19/2009 9:53 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>
> Hm, still nothing. Maybe I have to build it differently? I will play
> around with the environment settings, but any more input is
appreciated.
>
> -Turner
>
> On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
<[EMAIL PROTECTED]
> >wrote:
>
> > Don't point it at mapred-site.xml directly, but the conf directory,
> > instead.
> > (it is smart enough to pick up the new config files when it's
running on
> > 20).
> >
> > I tried setting the path directly to mapred-site.xml like you have
it,
> > and indeed, that doesn't work. Changing to the directory should fix
> > your issue.
> >
> > -D
> >
> > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]>
> wrote:
> > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> working
> > > with 0.18.0 and had it working fine.
> > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
since
> > Pig
> > > is looking for hadoop-site.xml instead of mapred-site.xml.
> > > My environment looks like this now:
> > >
> > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > >
> >
>
PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
conf/mapred-site.xml
> > > PIGDIR=/usr/local/pig-0.3.0
> > > HADOOP_HOME=/usr/local/hadoop
> > > HADOOPDIR=/usr/local/hadoop/conf
> > > PIG_HADOOP_VERSION=20
> > >
> > > Pig still isn't connecting correctly.
> > >
> > > -Turner
> > >
> > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
<[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> Nipun and Turner,
> > >> What are you setting PIG_CLASSPATH to?
> > >>
> > >> My environment works if I set it to
> > >> /path/to/pig.jar:path/to/mapred-site.xml
> > >>
> > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> > >> breakage -- I haven't quite decided if that's a bug or not.)
> > >>
> > >> For completeness, a full set of environment variables that make
it
> > >> work for me. I am running the Cloudera CDH2 testing package for
> > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> version
> > >> of Pig from trunk.
> > >>
> > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> > >> PIG_HADOOP_VERSION=20
> > >> PIG_CONF_DIR=./conf/
> > >>
> > >>
> >
>
PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
/pig.jar
> > >>
> > >>
> > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
Kunkel<[EMAIL PROTECTED]>
> > wrote:
> > >> > Hello,
> > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig
no
> > longer
> > >> > connects to the HDFS.
> > >> > I tried rebuilding and applying patch PIG660.
> > >> > I have a script that I run that exports these environment
tables
> > >> correctly:
> > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > >> > Pig still connects to file:/// instead of HDFS.
> > >> > I rebuilt the Pig package so that there are no patches applied
at
> the
> > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
from

Re: Pig 0.3.0 and Hadoop 0.20.0

Turner Kunkel 2009-08-19, 19:50

Ok, I know I'm really close now, I ran into this when running Pig on 18.
"Could not find the main class: org.apache.pig.Main"

I forgot how to resolve this.

Thanks a lot so far, by the way :)

-Turner

On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <[EMAIL PROTECTED]>wrote:

> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
> parameters and thing should start working
>
> -----Original Message-----
> From: Naber, Chad [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, August 19, 2009 11:52 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
>
> If you are not using Hadoop on Demand (hod), you will need to comment
> out the hod properties in pig's conf dir.
>
> -----Original Message-----
> From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, August 19, 2009 11:50 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>
> Getting closer...
> Actually got and error this time:
> ERROR 6012: Unable to run command: hod allocate ....
>
> -Turner
>
> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
> <[EMAIL PROTECTED]>wrote:
>
> > You need a special hadoop20.jar to build the pig sources. Check out
> PIG-660
> >
> > Santhosh
> >
> >
> > -----Original Message-----
> > From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> > Sent: Wed 8/19/2009 9:53 AM
> > To: [EMAIL PROTECTED]
> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >
> > Hm, still nothing. Maybe I have to build it differently? I will play
> > around with the environment settings, but any more input is
> appreciated.
> >
> > -Turner
> >
> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
> <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Don't point it at mapred-site.xml directly, but the conf directory,
> > > instead.
> > > (it is smart enough to pick up the new config files when it's
> running on
> > > 20).
> > >
> > > I tried setting the path directly to mapred-site.xml like you have
> it,
> > > and indeed, that doesn't work. Changing to the directory should fix
> > > your issue.
> > >
> > > -D
> > >
> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]>
> > wrote:
> > > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> > working
> > > > with 0.18.0 and had it working fine.
> > > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
> since
> > > Pig
> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
> > > > My environment looks like this now:
> > > >
> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > > >
> > >
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
> conf/mapred-site.xml
> > > > PIGDIR=/usr/local/pig-0.3.0
> > > > HADOOP_HOME=/usr/local/hadoop
> > > > HADOOPDIR=/usr/local/hadoop/conf
> > > > PIG_HADOOP_VERSION=20
> > > >
> > > > Pig still isn't connecting correctly.
> > > >
> > > > -Turner
> > > >
> > > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
> <[EMAIL PROTECTED]
> > > >wrote:
> > > >
> > > >> Nipun and Turner,
> > > >> What are you setting PIG_CLASSPATH to?
> > > >>
> > > >> My environment works if I set it to
> > > >> /path/to/pig.jar:path/to/mapred-site.xml
> > > >>
> > > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> > > >> breakage -- I haven't quite decided if that's a bug or not.)
> > > >>
> > > >> For completeness, a full set of environment variables that make
> it
> > > >> work for me. I am running the Cloudera CDH2 testing package for
> > > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> > version
> > > >> of Pig from trunk.
> > > >>
> > > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> > > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> > > >> PIG_HADOOP_VERSION=20
> > > >> PIG_CONF_DIR=./conf/
> > > >>
> > > >>
> > >
> >
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
> /pig.jar
> > > >>
> > > >>
> > > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
> Kunkel<[EMAIL PROTECTED]>
> > > wrote:
> > > >> > Hello,

Re: Pig 0.3.0 and Hadoop 0.20.0

Dmitriy Ryaboy 2009-08-19, 21:44

Tumer,
That error means you dropped pig.jar from your classpath.

On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> Ok, I know I'm really close now, I ran into this when running Pig on 18.
> "Could not find the main class: org.apache.pig.Main"
>
> I forgot how to resolve this.
>
> Thanks a lot so far, by the way :)
>
> -Turner
>
> On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <[EMAIL PROTECTED]>wrote:
>
>> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
>> parameters and thing should start working
>>
>> -----Original Message-----
>> From: Naber, Chad [mailto:[EMAIL PROTECTED]]
>> Sent: Wednesday, August 19, 2009 11:52 AM
>> To: [EMAIL PROTECTED]
>> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
>>
>> If you are not using Hadoop on Demand (hod), you will need to comment
>> out the hod properties in pig's conf dir.
>>
>> -----Original Message-----
>> From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
>> Sent: Wednesday, August 19, 2009 11:50 AM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>>
>> Getting closer...
>> Actually got and error this time:
>> ERROR 6012: Unable to run command: hod allocate ....
>>
>> -Turner
>>
>> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
>> <[EMAIL PROTECTED]>wrote:
>>
>> > You need a special hadoop20.jar to build the pig sources. Check out
>> PIG-660
>> >
>> > Santhosh
>> >
>> >
>> > -----Original Message-----
>> > From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
>> > Sent: Wed 8/19/2009 9:53 AM
>> > To: [EMAIL PROTECTED]
>> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>> >
>> > Hm, still nothing. Maybe I have to build it differently? I will play
>> > around with the environment settings, but any more input is
>> appreciated.
>> >
>> > -Turner
>> >
>> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
>> <[EMAIL PROTECTED]
>> > >wrote:
>> >
>> > > Don't point it at mapred-site.xml directly, but the conf directory,
>> > > instead.
>> > > (it is smart enough to pick up the new config files when it's
>> running on
>> > > 20).
>> > >
>> > > I tried setting the path directly to mapred-site.xml like you have
>> it,
>> > > and indeed, that doesn't work. Changing to the directory should fix
>> > > your issue.
>> > >
>> > > -D
>> > >
>> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]>
>> > wrote:
>> > > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
>> > working
>> > > > with 0.18.0 and had it working fine.
>> > > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
>> since
>> > > Pig
>> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
>> > > > My environment looks like this now:
>> > > >
>> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
>> > > >
>> > >
>> >
>> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
>> conf/mapred-site.xml
>> > > > PIGDIR=/usr/local/pig-0.3.0
>> > > > HADOOP_HOME=/usr/local/hadoop
>> > > > HADOOPDIR=/usr/local/hadoop/conf
>> > > > PIG_HADOOP_VERSION=20
>> > > >
>> > > > Pig still isn't connecting correctly.
>> > > >
>> > > > -Turner
>> > > >
>> > > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
>> <[EMAIL PROTECTED]
>> > > >wrote:
>> > > >
>> > > >> Nipun and Turner,
>> > > >> What are you setting PIG_CLASSPATH to?
>> > > >>
>> > > >> My environment works if I set it to
>> > > >> /path/to/pig.jar:path/to/mapred-site.xml
>> > > >>
>> > > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
>> > > >> breakage -- I haven't quite decided if that's a bug or not.)
>> > > >>
>> > > >> For completeness, a full set of environment variables that make
>> it
>> > > >> work for me. I am running the Cloudera CDH2 testing package for
>> > > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
>> > version
>> > > >> of Pig from trunk.
>> > > >>
>> > > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
>> > > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
>> > > >> PIG_HADOOP_VERSION=20

Re: Pig 0.3.0 and Hadoop 0.20.0

Turner Kunkel 2009-08-20, 13:29

It's finally working!

Using your environment table specs and
http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt, I got it to work.

Thanks much, this helps me a lot. Have a nice day.

-Turner

On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]>wrote:

> Tumer,
> That error means you dropped pig.jar from your classpath.
>
> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> > Ok, I know I'm really close now, I ran into this when running Pig on 18.
> > "Could not find the main class: org.apache.pig.Main"
> >
> > I forgot how to resolve this.
> >
> > Thanks a lot so far, by the way :)
> >
> > -Turner
> >
> > On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <[EMAIL PROTECTED]
> >wrote:
> >
> >> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
> >> parameters and thing should start working
> >>
> >> -----Original Message-----
> >> From: Naber, Chad [mailto:[EMAIL PROTECTED]]
> >> Sent: Wednesday, August 19, 2009 11:52 AM
> >> To: [EMAIL PROTECTED]
> >> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
> >>
> >> If you are not using Hadoop on Demand (hod), you will need to comment
> >> out the hod properties in pig's conf dir.
> >>
> >> -----Original Message-----
> >> From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> >> Sent: Wednesday, August 19, 2009 11:50 AM
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >>
> >> Getting closer...
> >> Actually got and error this time:
> >> ERROR 6012: Unable to run command: hod allocate ....
> >>
> >> -Turner
> >>
> >> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
> >> <[EMAIL PROTECTED]>wrote:
> >>
> >> > You need a special hadoop20.jar to build the pig sources. Check out
> >> PIG-660
> >> >
> >> > Santhosh
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> >> > Sent: Wed 8/19/2009 9:53 AM
> >> > To: [EMAIL PROTECTED]
> >> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >> >
> >> > Hm, still nothing. Maybe I have to build it differently? I will play
> >> > around with the environment settings, but any more input is
> >> appreciated.
> >> >
> >> > -Turner
> >> >
> >> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
> >> <[EMAIL PROTECTED]
> >> > >wrote:
> >> >
> >> > > Don't point it at mapred-site.xml directly, but the conf directory,
> >> > > instead.
> >> > > (it is smart enough to pick up the new config files when it's
> >> running on
> >> > > 20).
> >> > >
> >> > > I tried setting the path directly to mapred-site.xml like you have
> >> it,
> >> > > and indeed, that doesn't work. Changing to the directory should fix
> >> > > your issue.
> >> > >
> >> > > -D
> >> > >
> >> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]>
> >> > wrote:
> >> > > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> >> > working
> >> > > > with 0.18.0 and had it working fine.
> >> > > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
> >> since
> >> > > Pig
> >> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
> >> > > > My environment looks like this now:
> >> > > >
> >> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> >> > > >
> >> > >
> >> >
> >> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
> >> conf/mapred-site.xml
> >> > > > PIGDIR=/usr/local/pig-0.3.0
> >> > > > HADOOP_HOME=/usr/local/hadoop
> >> > > > HADOOPDIR=/usr/local/hadoop/conf
> >> > > > PIG_HADOOP_VERSION=20
> >> > > >
> >> > > > Pig still isn't connecting correctly.
> >> > > >
> >> > > > -Turner
> >> > > >
> >> > > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
> >> <[EMAIL PROTECTED]
> >> > > >wrote:
> >> > > >
> >> > > >> Nipun and Turner,
> >> > > >> What are you setting PIG_CLASSPATH to?
> >> > > >>
> >> > > >> My environment works if I set it to
> >> > > >> /path/to/pig.jar:path/to/mapred-site.xml
> >> > > >>
> >> > > >> (leaving off the path to mapred-site.xml or pig.jar both lead to

Re: Pig 0.3.0 and Hadoop 0.20.0

Dmitriy Ryaboy 2009-08-20, 16:38

Turner,
Before you went through those steps, did you use a vanilla hadoop 20
jar or did you get one from the zebra ticket?
I am still at a loss as to why it wasn't working for you before.
-D

On Thu, Aug 20, 2009 at 6:29 AM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> It's finally working!
>
> Using your environment table specs and
> http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt, I got it to work.
>
> Thanks much, this helps me a lot. Have a nice day.
>
> -Turner
>
> On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]>wrote:
>
>> Tumer,
>> That error means you dropped pig.jar from your classpath.
>>
>> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
>> > Ok, I know I'm really close now, I ran into this when running Pig on 18.
>> > "Could not find the main class: org.apache.pig.Main"
>> >
>> > I forgot how to resolve this.
>> >
>> > Thanks a lot so far, by the way :)
>> >
>> > -Turner
>> >
>> > On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
>> >> parameters and thing should start working
>> >>
>> >> -----Original Message-----
>> >> From: Naber, Chad [mailto:[EMAIL PROTECTED]]
>> >> Sent: Wednesday, August 19, 2009 11:52 AM
>> >> To: [EMAIL PROTECTED]
>> >> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
>> >>
>> >> If you are not using Hadoop on Demand (hod), you will need to comment
>> >> out the hod properties in pig's conf dir.
>> >>
>> >> -----Original Message-----
>> >> From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
>> >> Sent: Wednesday, August 19, 2009 11:50 AM
>> >> To: [EMAIL PROTECTED]
>> >> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>> >>
>> >> Getting closer...
>> >> Actually got and error this time:
>> >> ERROR 6012: Unable to run command: hod allocate ....
>> >>
>> >> -Turner
>> >>
>> >> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
>> >> <[EMAIL PROTECTED]>wrote:
>> >>
>> >> > You need a special hadoop20.jar to build the pig sources. Check out
>> >> PIG-660
>> >> >
>> >> > Santhosh
>> >> >
>> >> >
>> >> > -----Original Message-----
>> >> > From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
>> >> > Sent: Wed 8/19/2009 9:53 AM
>> >> > To: [EMAIL PROTECTED]
>> >> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>> >> >
>> >> > Hm, still nothing. Maybe I have to build it differently? I will play
>> >> > around with the environment settings, but any more input is
>> >> appreciated.
>> >> >
>> >> > -Turner
>> >> >
>> >> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
>> >> <[EMAIL PROTECTED]
>> >> > >wrote:
>> >> >
>> >> > > Don't point it at mapred-site.xml directly, but the conf directory,
>> >> > > instead.
>> >> > > (it is smart enough to pick up the new config files when it's
>> >> running on
>> >> > > 20).
>> >> > >
>> >> > > I tried setting the path directly to mapred-site.xml like you have
>> >> it,
>> >> > > and indeed, that doesn't work. Changing to the directory should fix
>> >> > > your issue.
>> >> > >
>> >> > > -D
>> >> > >
>> >> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<[EMAIL PROTECTED]>
>> >> > wrote:
>> >> > > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
>> >> > working
>> >> > > > with 0.18.0 and had it working fine.
>> >> > > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
>> >> since
>> >> > > Pig
>> >> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
>> >> > > > My environment looks like this now:
>> >> > > >
>> >> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
>> >> > > >
>> >> > >
>> >> >
>> >> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
>> >> conf/mapred-site.xml
>> >> > > > PIGDIR=/usr/local/pig-0.3.0
>> >> > > > HADOOP_HOME=/usr/local/hadoop
>> >> > > > HADOOPDIR=/usr/local/hadoop/conf
>> >> > > > PIG_HADOOP_VERSION=20
>> >> > > >
>> >> > > > Pig still isn't connecting correctly.
>> >> > > >
>> >> > > > -Turner

Re: Pig 0.3.0 and Hadoop 0.20.0

Turner Kunkel 2009-08-20, 16:52

I built hadoop20.jar using ant after patching the Hadoop 0.20.0
download with the patch provided on Alex's how-to site.
I grabbed Pig using svn and patched it with PIG-660_4.patch, then
moved hadoop20.jar into Pig's lib directory, then built pig.jar with
ant.
I set my classpath to include the hadoop conf folder and pig.jar
(along with the other environment settings).
I execute pig with this command:

java -Dfs.default.name=hdfs://localhost:9000
-Dmapred.job.tracker=localhost:9001 -jar pig.jar (with my own hdfs
and mapred settings of course)

So I was doing multiple things differently after coming from Hadoop
18, but this seems to work.

-Turner

On Thu, Aug 20, 2009 at 11:38 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:
>
> Turner,
> Before you went through those steps, did you use a vanilla hadoop 20
> jar or did you get one from the zebra ticket?
> I am still at a loss as to why it wasn't working for you before.
> -D
>
> On Thu, Aug 20, 2009 at 6:29 AM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> > It's finally working!
> >
> > Using your environment table specs and
> > http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt, I got it to work.
> >
> > Thanks much, this helps me a lot. Have a nice day.
> >
> > -Turner
> >
> > On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]>wrote:
> >
> >> Tumer,
> >> That error means you dropped pig.jar from your classpath.
> >>
> >> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<[EMAIL PROTECTED]> wrote:
> >> > Ok, I know I'm really close now, I ran into this when running Pig on 18.
> >> > "Could not find the main class: org.apache.pig.Main"
> >> >
> >> > I forgot how to resolve this.
> >> >
> >> > Thanks a lot so far, by the way :)
> >> >
> >> > -Turner
> >> >
> >> > On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <[EMAIL PROTECTED]
> >> >wrote:
> >> >
> >> >> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
> >> >> parameters and thing should start working
> >> >>
> >> >> -----Original Message-----
> >> >> From: Naber, Chad [mailto:[EMAIL PROTECTED]]
> >> >> Sent: Wednesday, August 19, 2009 11:52 AM
> >> >> To: [EMAIL PROTECTED]
> >> >> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
> >> >>
> >> >> If you are not using Hadoop on Demand (hod), you will need to comment
> >> >> out the hod properties in pig's conf dir.
> >> >>
> >> >> -----Original Message-----
> >> >> From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> >> >> Sent: Wednesday, August 19, 2009 11:50 AM
> >> >> To: [EMAIL PROTECTED]
> >> >> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >> >>
> >> >> Getting closer...
> >> >> Actually got and error this time:
> >> >> ERROR 6012: Unable to run command: hod allocate ....
> >> >>
> >> >> -Turner
> >> >>
> >> >> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
> >> >> <[EMAIL PROTECTED]>wrote:
> >> >>
> >> >> > You need a special hadoop20.jar to build the pig sources. Check out
> >> >> PIG-660
> >> >> >
> >> >> > Santhosh
> >> >> >
> >> >> >
> >> >> > -----Original Message-----
> >> >> > From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> >> >> > Sent: Wed 8/19/2009 9:53 AM
> >> >> > To: [EMAIL PROTECTED]
> >> >> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >> >> >
> >> >> > Hm, still nothing. Maybe I have to build it differently? I will play
> >> >> > around with the environment settings, but any more input is
> >> >> appreciated.
> >> >> >
> >> >> > -Turner
> >> >> >
> >> >> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
> >> >> <[EMAIL PROTECTED]
> >> >> > >wrote:
> >> >> >
> >> >> > > Don't point it at mapred-site.xml directly, but the conf directory,
> >> >> > > instead.
> >> >> > > (it is smart enough to pick up the new config files when it's
> >> >> running on
> >> >> > > 20).
> >> >> > >
> >> >> > > I tried setting the path directly to mapred-site.xml like you have
> >> >> it,
> >> >> > > and indeed, that doesn't work. Changing to the directory should fix
> >> >> > > your issue.
> >> >> > >

Re: Pig 0.3.0 and Hadoop 0.20.0

George Pang 2009-09-04, 20:15

Hi Turner,

I use Alex's tutorial on building Pig with Ant but get errors like this,

........
er.java:36: package org.apache.commons.logging does not exist
   [javac] import org.apache.commons.logging.LogFactory;
   [javac] ^
   [javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:40:
package org.apache.commons.logging does not exist
   [javac] import org.apache.commons.logging.Log;
   [javac] ^
   [javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:41:
package org.apache.commons.logging does not exist
   [javac] import org.apache.commons.logging.LogFactory;
   [javac] ^
   [javac] Note: Some input files use unchecked or unsafe operations.
   [javac] Note: Recompile with -Xlint:unchecked for details.
   [javac] 100 errors

BUILD FAILED
/home/hadoop20/pig/build.xml:246: The following error occurred while
executing this line:
/home/hadoop20/pig/build.xml:285: Compile failed; see the compiler error
output for details.

Where did I get wrong? Or where to find the error output file?

Thank you.

George

2009/8/20 Turner Kunkel <[EMAIL PROTECTED]>

> I built hadoop20.jar using ant after patching the Hadoop 0.20.0
> download with the patch provided on Alex's how-to site.
> I grabbed Pig using svn and patched it with PIG-660_4.patch, then
> moved hadoop20.jar into Pig's lib directory, then built pig.jar with
> ant.
> I set my classpath to include the hadoop conf folder and pig.jar
> (along with the other environment settings).
> I execute pig with this command:
>
> java -Dfs.default.name=hdfs://localhost:9000
> -Dmapred.job.tracker=localhost:9001 -jar pig.jar (with my own hdfs
> and mapred settings of course)
>
> So I was doing multiple things differently after coming from Hadoop
> 18, but this seems to work.
>
> -Turner
>
> On Thu, Aug 20, 2009 at 11:38 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]>
> wrote:
> >
> > Turner,
> > Before you went through those steps, did you use a vanilla hadoop 20
> > jar or did you get one from the zebra ticket?
> > I am still at a loss as to why it wasn't working for you before.
> > -D
> >
> > On Thu, Aug 20, 2009 at 6:29 AM, Turner Kunkel<[EMAIL PROTECTED]>
> wrote:
> > > It's finally working!
> > >
> > > Using your environment table specs and
> > > http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt<http://behemoth.strlen.net/%7Ealex/hadoop20-pig-howto.txt>,
> I got it to work.
> > >
> > > Thanks much, this helps me a lot. Have a nice day.
> > >
> > > -Turner
> > >
> > > On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]
> >wrote:
> > >
> > >> Tumer,
> > >> That error means you dropped pig.jar from your classpath.
> > >>
> > >> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<[EMAIL PROTECTED]>
> wrote:
> > >> > Ok, I know I'm really close now, I ran into this when running Pig on
> 18.
> > >> > "Could not find the main class: org.apache.pig.Main"
> > >> >
> > >> > I forgot how to resolve this.
> > >> >
> > >> > Thanks a lot so far, by the way :)
> > >> >
> > >> > -Turner
> > >> >
> > >> > On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <
> [EMAIL PROTECTED]
> > >> >wrote:
> > >> >
> > >> >> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the
> hod
> > >> >> parameters and thing should start working
> > >> >>
> > >> >> -----Original Message-----
> > >> >> From: Naber, Chad [mailto:[EMAIL PROTECTED]]
> > >> >> Sent: Wednesday, August 19, 2009 11:52 AM
> > >> >> To: [EMAIL PROTECTED]
> > >> >> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
> > >> >>
> > >> >> If you are not using Hadoop on Demand (hod), you will need to
> comment
> > >> >> out the hod properties in pig's conf dir.
> > >> >>
> > >> >> -----Original Message-----
> > >> >> From: Turner Kunkel [mailto:[EMAIL PROTECTED]]
> > >> >> Sent: Wednesday, August 19, 2009 11:50 AM
> > >> >> To: [EMAIL PROTECTED]
> > >> >> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0

Re: Pig 0.3.0 and Hadoop 0.20.0

Turner Kunkel 2009-09-04, 20:46

Did you patch using PIG-660_4.patch?

It looks like it's not looking in the correct spot for Java classes.
Did you download everything from the svn Pig package correctly?

I also ran into the issue of running ant with certain permissions. I
dedicated full rights to user 'hadoop' on all of my Hadoop folders
(pig, hadoop, etc.) and ant only succeeded when I was logged in as
user 'hadoop' and ran ant.

-Turner

On Fri, Sep 4, 2009 at 3:15 PM, George Pang<[EMAIL PROTECTED]> wrote:
> Hi Turner,
>
> I use Alex's tutorial on building Pig with Ant but get errors like this,
>
> ........
> er.java:36: package org.apache.commons.logging does not exist
> [javac] import org.apache.commons.logging.LogFactory;
> [javac] ^
> [javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:40:
> package org.apache.commons.logging does not exist
> [javac] import org.apache.commons.logging.Log;
> [javac] ^
> [javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:41:
> package org.apache.commons.logging does not exist
> [javac] import org.apache.commons.logging.LogFactory;
> [javac] ^
> [javac] Note: Some input files use unchecked or unsafe operations.
> [javac] Note: Recompile with -Xlint:unchecked for details.
> [javac] 100 errors
>
> BUILD FAILED
> /home/hadoop20/pig/build.xml:246: The following error occurred while
> executing this line:
> /home/hadoop20/pig/build.xml:285: Compile failed; see the compiler error
> output for details.
>
> Where did I get wrong? Or where to find the error output file?
>
> Thank you.
>
> George
>
>
>
> 2009/8/20 Turner Kunkel <[EMAIL PROTECTED]>
>
>> I built hadoop20.jar using ant after patching the Hadoop 0.20.0
>> download with the patch provided on Alex's how-to site.
>> I grabbed Pig using svn and patched it with PIG-660_4.patch, then
>> moved hadoop20.jar into Pig's lib directory, then built pig.jar with
>> ant.
>> I set my classpath to include the hadoop conf folder and pig.jar
>> (along with the other environment settings).
>> I execute pig with this command:
>>
>> java -Dfs.default.name=hdfs://localhost:9000
>> -Dmapred.job.tracker=localhost:9001 -jar pig.jar (with my own hdfs
>> and mapred settings of course)
>>
>> So I was doing multiple things differently after coming from Hadoop
>> 18, but this seems to work.
>>
>> -Turner
>>
>> On Thu, Aug 20, 2009 at 11:38 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]>
>> wrote:
>> >
>> > Turner,
>> > Before you went through those steps, did you use a vanilla hadoop 20
>> > jar or did you get one from the zebra ticket?
>> > I am still at a loss as to why it wasn't working for you before.
>> > -D
>> >
>> > On Thu, Aug 20, 2009 at 6:29 AM, Turner Kunkel<[EMAIL PROTECTED]>
>> wrote:
>> > > It's finally working!
>> > >
>> > > Using your environment table specs and
>> > > http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt<http://behemoth.strlen.net/%7Ealex/hadoop20-pig-howto.txt>,
>> I got it to work.
>> > >
>> > > Thanks much, this helps me a lot. Have a nice day.
>> > >
>> > > -Turner
>> > >
>> > > On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]
>> >wrote:
>> > >
>> > >> Tumer,
>> > >> That error means you dropped pig.jar from your classpath.
>> > >>
>> > >> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<[EMAIL PROTECTED]>
>> wrote:
>> > >> > Ok, I know I'm really close now, I ran into this when running Pig on
>> 18.
>> > >> > "Could not find the main class: org.apache.pig.Main"
>> > >> >
>> > >> > I forgot how to resolve this.
>> > >> >
>> > >> > Thanks a lot so far, by the way :)
>> > >> >
>> > >> > -Turner
>> > >> >
>> > >> > On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <
>> [EMAIL PROTECTED]
>> > >> >wrote:
>> > >> >
>> > >> >> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the
>> hod
>> > >> >> parameters and thing should start working
>> > >> >>
>> > >> >> -----Original Message-----

Re: Pig 0.3.0 and Hadoop 0.20.0

George Pang 2009-09-05, 00:54

Yes, I use PIG-660_4.patch. And I check out from svn.

I will try again. Thank you.

George

2009/9/4 Turner Kunkel <[EMAIL PROTECTED]>

> Did you patch using PIG-660_4.patch?
>
> It looks like it's not looking in the correct spot for Java classes.
> Did you download everything from the svn Pig package correctly?
>
> I also ran into the issue of running ant with certain permissions. I
> dedicated full rights to user 'hadoop' on all of my Hadoop folders
> (pig, hadoop, etc.) and ant only succeeded when I was logged in as
> user 'hadoop' and ran ant.
>
> -Turner
>
> On Fri, Sep 4, 2009 at 3:15 PM, George Pang<[EMAIL PROTECTED]> wrote:
> > Hi Turner,
> >
> > I use Alex's tutorial on building Pig with Ant but get errors like this,
> >
> > ........
> > er.java:36: package org.apache.commons.logging does not exist
> > [javac] import org.apache.commons.logging.LogFactory;
> > [javac] ^
> > [javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:40:
> > package org.apache.commons.logging does not exist
> > [javac] import org.apache.commons.logging.Log;
> > [javac] ^
> > [javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:41:
> > package org.apache.commons.logging does not exist
> > [javac] import org.apache.commons.logging.LogFactory;
> > [javac] ^
> > [javac] Note: Some input files use unchecked or unsafe operations.
> > [javac] Note: Recompile with -Xlint:unchecked for details.
> > [javac] 100 errors
> >
> > BUILD FAILED
> > /home/hadoop20/pig/build.xml:246: The following error occurred while
> > executing this line:
> > /home/hadoop20/pig/build.xml:285: Compile failed; see the compiler error
> > output for details.
> >
> > Where did I get wrong? Or where to find the error output file?
> >
> > Thank you.
> >
> > George
> >
> >
> >
> > 2009/8/20 Turner Kunkel <[EMAIL PROTECTED]>
> >
> >> I built hadoop20.jar using ant after patching the Hadoop 0.20.0
> >> download with the patch provided on Alex's how-to site.
> >> I grabbed Pig using svn and patched it with PIG-660_4.patch, then
> >> moved hadoop20.jar into Pig's lib directory, then built pig.jar with
> >> ant.
> >> I set my classpath to include the hadoop conf folder and pig.jar
> >> (along with the other environment settings).
> >> I execute pig with this command:
> >>
> >> java -Dfs.default.name=hdfs://localhost:9000
> >> -Dmapred.job.tracker=localhost:9001 -jar pig.jar (with my own hdfs
> >> and mapred settings of course)
> >>
> >> So I was doing multiple things differently after coming from Hadoop
> >> 18, but this seems to work.
> >>
> >> -Turner
> >>
> >> On Thu, Aug 20, 2009 at 11:38 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]
> >
> >> wrote:
> >> >
> >> > Turner,
> >> > Before you went through those steps, did you use a vanilla hadoop 20
> >> > jar or did you get one from the zebra ticket?
> >> > I am still at a loss as to why it wasn't working for you before.
> >> > -D
> >> >
> >> > On Thu, Aug 20, 2009 at 6:29 AM, Turner Kunkel<[EMAIL PROTECTED]>
> >> wrote:
> >> > > It's finally working!
> >> > >
> >> > > Using your environment table specs and
> >> > > http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt<http://behemoth.strlen.net/%7Ealex/hadoop20-pig-howto.txt>
> <http://behemoth.strlen.net/%7Ealex/hadoop20-pig-howto.txt>,
> >> I got it to work.
> >> > >
> >> > > Thanks much, this helps me a lot. Have a nice day.
> >> > >
> >> > > -Turner
> >> > >
> >> > > On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <
> [EMAIL PROTECTED]
> >> >wrote:
> >> > >
> >> > >> Tumer,
> >> > >> That error means you dropped pig.jar from your classpath.
> >> > >>
> >> > >> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<[EMAIL PROTECTED]
> >
> >> wrote:
> >> > >> > Ok, I know I'm really close now, I ran into this when running Pig
> on
> >> 18.
> >> > >> > "Could not find the main class: org.apache.pig.Main"
> >> > >> >
> >>