Hello,

I seem to have broken my Pig install, and I don't know where to look.

If I use directly the script (grunt) everything works ok, but every time I
try to run a pig script:  'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
org.apache.pig.Main script1-hadoop.pig'

I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Undefined parameter : PIGDIR

Obviously I have the PIGDIR var set:
> echo $PIGDIR
> /home/alex/hadoop/pig

This is something that I did, as I have used 0.5 and 0.6 and a patched
version of 0.6 in parallel, but I can't figure out where to look. Any
version I try to start now, gives the same error.

any help would be greatly appreciated!

alex

what does your script1-hadoop.pig look like?

The error you are getting happens when the pig preprocessor can't
substitute Pig variables (the stuff you specify with -param and
%default, etc).  Do you have $PIGDIR in your script somewhere?

-D

On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
<[EMAIL PROTECTED]> wrote:
> Hello,
>
> I seem to have broken my Pig install, and I don't know where to look.
>
> If I use directly the script (grunt) everything works ok, but every time I
> try to run a pig script:  'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
> org.apache.pig.Main script1-hadoop.pig'
>
> I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
> Unexpected internal error. Undefined parameter : PIGDIR
>
> Obviously I have the PIGDIR var set:
>> echo $PIGDIR
>> /home/alex/hadoop/pig
>
> This is something that I did, as I have used 0.5 and 0.6 and a patched
> version of 0.6 in parallel, but I can't figure out where to look. Any
> version I try to start now, gives the same error.
>
> any help would be greatly appreciated!
>
> alex
>

Hello,

sorry for the delay, but I wanted to build from the source again just to
make sure.

The script is like this:

'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as (id:
long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4: int); '

That's it. My guess is that Pig doesn't even start. Do you think I need to
change something in the properties file? I'm not sure anymore and I don't
have a lot of luck going through the sources.

And another thing, I tried running 'ant test' for both 0.6-branch and trunk
at the same time (because they take a very long time) and both test scripts
failed. I've switched to running them one after the other and they are fine.
Do you think that is ok?

thanks for your time,
alex

On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> what does your script1-hadoop.pig look like?
>
> The error you are getting happens when the pig preprocessor can't
> substitute Pig variables (the stuff you specify with -param and
> %default, etc).  Do you have $PIGDIR in your script somewhere?
>
> -D
>
> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
> <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> > I seem to have broken my Pig install, and I don't know where to look.
> >
> > If I use directly the script (grunt) everything works ok, but every time
> I
> > try to run a pig script:  'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
> > org.apache.pig.Main script1-hadoop.pig'
> >
> > I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
> > Unexpected internal error. Undefined parameter : PIGDIR
> >
> > Obviously I have the PIGDIR var set:
> >> echo $PIGDIR
> >> /home/alex/hadoop/pig
> >
> > This is something that I did, as I have used 0.5 and 0.6 and a patched
> > version of 0.6 in parallel, but I can't figure out where to look. Any
> > version I try to start now, gives the same error.
> >
> > any help would be greatly appreciated!
> >
> > alex
> >
>

Pig starts, the error you are getting is from inside pig.Main .
I think something is getting messed up in your environment because you
are juggling too many different versions of pig (granted, I have 3 or
4 in various stages of development on my laptop most of the time, and
haven't had your problems. But then neither of them has a hacked
bin/pig , with the exception of the cloudera one...). I've never tried
running multiple ant tests, either. There's a short "sanity" version
of tests, ant test-commit, that runs in under 10 minutes. You might
want to try that if you are not doing things like changing join
implementations or messing with the optimizer.

Let's do this: send a full trace of what you are doing and what your
environment looks like.  Something like

pwd
printenv | grep JAVA
printenv | grep PIG
printenv | grep HADOOP
cat conf/pig.properties
java -cp ........
<boom!>

-D
On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
<[EMAIL PROTECTED]> wrote:
> Hello,
>
> sorry for the delay, but I wanted to build from the source again just to
> make sure.
>
> The script is like this:
>
> 'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as (id:
> long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4: int); '
>
> That's it. My guess is that Pig doesn't even start. Do you think I need to
> change something in the properties file? I'm not sure anymore and I don't
> have a lot of luck going through the sources.
>
> And another thing, I tried running 'ant test' for both 0.6-branch and trunk
> at the same time (because they take a very long time) and both test scripts
> failed. I've switched to running them one after the other and they are fine.
> Do you think that is ok?
>
> thanks for your time,
> alex
>
> On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:
>
>> what does your script1-hadoop.pig look like?
>>
>> The error you are getting happens when the pig preprocessor can't
>> substitute Pig variables (the stuff you specify with -param and
>> %default, etc).  Do you have $PIGDIR in your script somewhere?
>>
>> -D
>>
>> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
>> <[EMAIL PROTECTED]> wrote:
>> > Hello,
>> >
>> > I seem to have broken my Pig install, and I don't know where to look.
>> >
>> > If I use directly the script (grunt) everything works ok, but every time
>> I
>> > try to run a pig script:  'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
>> > org.apache.pig.Main script1-hadoop.pig'
>> >
>> > I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
>> > Unexpected internal error. Undefined parameter : PIGDIR
>> >
>> > Obviously I have the PIGDIR var set:
>> >> echo $PIGDIR
>> >> /home/alex/hadoop/pig
>> >
>> > This is something that I did, as I have used 0.5 and 0.6 and a patched
>> > version of 0.6 in parallel, but I can't figure out where to look. Any
>> > version I try to start now, gives the same error.
>> >
>> > any help would be greatly appreciated!
>> >
>> > alex
>> >
>>
>

Hello

And thanks again for all your help

I have a symbolic link /home/alex/hadoop/pig which points to
/home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of the
0.6 branch.
I'm running the script from /home/alex/hadoop/test.

I didn't touch pig.properties, also I'm running with the default pig script.

The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not work(Exception
in thread "main" java.lang.NoClassDefFoundError: org/apache/pig/Main).

I'm running with a fresh 0.6 build and still no luck.

The info you reqested:

alex@alex-desktop:~/hadoop/test$ pwd
/home/alex/hadoop/test
alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
JAVA_HOME=/usr/lib/jvm/java-6-sun
alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
PIGDIR=/home/alex/hadoop/pig
alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
HADOOP_HOME=/home/alex/hadoop/hadoop
HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
HADOOPDIR=/home/alex/hadoop/hadoop/conf

alex@alex-desktop:~/hadoop/test$ cat
/home/alex/hadoop/pig/conf/pig.properties
# Pig configuration file. All values can be overwritten by command line
arguments.
# see bin/pig -help

# log4jconf log4j configuration file
# log4jconf=./conf/log4j.properties

# brief logging (no timestamps)
brief=false

# clustername, name of the hadoop jobtracker. If no port is defined port
50020 will be used.
#cluster

#debug level, INFO is default
debug=INFO

# a file that contains pig script
#file
# load jarfile, colon separated
#jar
#verbose print all log messages to screen (default to print only INFO and
above to screen)
verbose=false

#exectype local|mapreduce, mapreduce is default
#exectype=mapreduce
# hod realted properties
#ssh.gateway
#hod.expect.root
#hod.expect.uselatest
#hod.command
#hod.config.dir
#hod.param
#Do not spill temp files smaller than this size (bytes)
pig.spill.size.threshold=5000000
#EXPERIMENT: Activate garbage collection when spilling a file bigger than
this size (bytes)
#This should help reduce the number of files being spilled.
pig.spill.gc.activation.size=40000000
######################
# Everything below this line is Yahoo specific.  Note that I've made
# (almost) no changes to the lines above to make merging in from Apache
# easier.  Any values I don't want from above I override below.
#
# This file is configured for use with HOD on the production clusters.  If
you
# want to run pig with a static cluster you will need to remove everything
# below this line and set the cluster value (above) to the
# hostname and port of your job tracker.

exectype=mapreduce

hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
hod.server=local

cluster.domain=inktomisearch.com

log.file
yinst.cluster=kryptonite

And now boom!

java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR org.apache.pig.Main
test_lite.pig
2010-02-16 12:27:17,283 [main] INFO  org.apache.pig.Main - Logging error
messages to: /home/alex/hadoop/test/pig_1266319637282.log
2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Undefined parameter : PIGDIR
Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log

The log file:
alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR

java.lang.RuntimeException: Undefined parameter : PIGDIR
    at
org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
    at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
    at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
    at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
    at org.apache.pig.Main.main(Main.java:366)
===============================================================================

The script: 'A = load '/home/alex/hadoop/test/t.csv' using PigStorage('\t')
as (id: long); '
The file: '1' .
Simple enough :)

I'm all out of ideas. It seems that even 0.5 is broken now. I can't start
anything as scripts. If I go with manual processing in grunt (line by line)
it's all good on any version.

thanks,
alex
On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

Hm, nothing jumps out. Definitely the error you are getting indicates that
somehow the preprocessor is trying to substitute a variable called PIGDIR in
your script. Which is odd. Does the same thing happen if you try bin/pig -f
test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's secret),
and sending along the output. Why are you using pig-core instead of pig.jar
that ant should be generating for you when you build? Where did you get it,
how did you build it, and what's the output of cksum on it?
The fact that now your 0.5 is broken too makes me think that maybe your
symlinks are messed up. Surely a change to one jar shouldn't be affecting a
totally unrelated jar in a directory the first jar doesn't know about.

Sorry about the barrage of questions, I am just a bit dumbfounded about how
this could even begin to start happening. Any preprocessor experts around?

-D
On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
<[EMAIL PROTECTED]>wrote:

> Hello
>
> And thanks again for all your help
>
> I have a symbolic link /home/alex/hadoop/pig which points to
> /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of the
> 0.6 branch.
> I'm running the script from /home/alex/hadoop/test.
>
> I didn't touch pig.properties, also I'm running with the default pig
> script.
>
> The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
> to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
> work(Exception
> in thread "main" java.lang.NoClassDefFoundError: org/apache/pig/Main).
>
> I'm running with a fresh 0.6 build and still no luck.
>
> The info you reqested:
>
> alex@alex-desktop:~/hadoop/test$ pwd
> /home/alex/hadoop/test
> alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
> JAVA_HOME=/usr/lib/jvm/java-6-sun
> alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
> PIGDIR=/home/alex/hadoop/pig
> alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
> HADOOP_HOME=/home/alex/hadoop/hadoop
> HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> HADOOPDIR=/home/alex/hadoop/hadoop/conf
>
> alex@alex-desktop:~/hadoop/test$ cat
> /home/alex/hadoop/pig/conf/pig.properties
> # Pig configuration file. All values can be overwritten by command line
> arguments.
> # see bin/pig -help
>
> # log4jconf log4j configuration file
> # log4jconf=./conf/log4j.properties
>
> # brief logging (no timestamps)
> brief=false
>
> # clustername, name of the hadoop jobtracker. If no port is defined port
> 50020 will be used.
> #cluster
>
> #debug level, INFO is default
> debug=INFO
>
> # a file that contains pig script
> #file>
> # load jarfile, colon separated
> #jar>
> #verbose print all log messages to screen (default to print only INFO and
> above to screen)
> verbose=false
>
> #exectype local|mapreduce, mapreduce is default
> #exectype=mapreduce
> # hod realted properties
> #ssh.gateway
> #hod.expect.root
> #hod.expect.uselatest
> #hod.command
> #hod.config.dir
> #hod.param
>
>
> #Do not spill temp files smaller than this size (bytes)
> pig.spill.size.threshold=5000000
> #EXPERIMENT: Activate garbage collection when spilling a file bigger than
> this size (bytes)
> #This should help reduce the number of files being spilled.
> pig.spill.gc.activation.size=40000000
>
>
> ######################
> # Everything below this line is Yahoo specific.  Note that I've made
> # (almost) no changes to the lines above to make merging in from Apache
> # easier.  Any values I don't want from above I override below.
> #
> # This file is configured for use with HOD on the production clusters.  If
> you
> # want to run pig with a static cluster you will need to remove everything
> # below this line and set the cluster value (above) to the
> # hostname and port of your job tracker.
>
> exectype=mapreduce
>
> hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
> hod.server=local
>
> cluster.domain=inktomisearch.com
>
> log.file>
> yinst.cluster=kryptonite
>
>
>
> And now boom!
>
> java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR org.apache.pig.Main

Hello

This is getting tiresome :)

Today, I tried with a fresh download of pig 0.5 and I get the exact same
error. Something somewhere just broke down, and it took my machine with it.
At this point I have no way to run any kind of pig version.
Needless to say, this is a big problem for me :)

Another thing I tried is to do a fresh build of the 0.6 branch. Same error.
I even dropped the sym links.

Back to our debugging session:

> svn co http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2

Fetching external item into 'pig-branch-0.6-take2/test/bin'
A    pig-branch-0.6-take2/test/bin/test-patch.sh
Checked out external at revision 910901.

Checked out revision 910900.

> cd pig-branch-0.6-take2/
> ant
BUILD SUCCESSFUL
Total time: 26 seconds

> cd bin/

next is to check if all vars are there
> printenv | grep PIG
PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2

> printenv | grep HADOOP
HADOOP_HOME=/home/alex/hadoop/hadoop
HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
HADOOPDIR=/home/alex/hadoop/hadoop/conf

> pig
Exception in thread "main" java.lang.NoClassDefFoundError:
jline/ConsoleReaderInputStream
Caused by: java.lang.ClassNotFoundException: jline.ConsoleReaderInputStream
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: org.apache.pig.Main.  Program will exit.

I've sent some email to the list about this problem before. A quick fix is
to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
> cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar

> pig
2010-02-17 11:48:52,843 [main] INFO  org.apache.pig.Main - Logging error
messages to:
/home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
2010-02-17 11:48:53,175 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: file:///
grunt>

Now at least it will start.

Getting to the initial issue:

the '-f' option:
> pig -f /home/alex/hadoop/test/test_lite.pig
2010-02-17 11:49:28,988 [main] INFO  org.apache.pig.Main - Logging error
messages to:
/home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Undefined parameter : PIGDIR
Details at logfile:
/home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log

the '- secretDebugCmd' option

pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
dry run:
/usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
-Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
-Dpig.log.file=pig.log
-Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
-Dpig.root.logger=INFO,console,DRFA -classpath
/home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar
org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig

I did not change the scripts, and I get the same error.

This is really killing me :)

thanks,
alex

On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> Hm, nothing jumps out. Definitely the error you are getting indicates that
> somehow the preprocessor is trying to substitute a variable called PIGDIR
> in
> your script. Which is odd. Does the same thing happen if you try bin/pig -f

Probably unrelated, but I always run from the root directory, not from bin/,
when working with an svn version. Also, never had dev vs core issues. Can
you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
pig.jar (not core, not  dev) on $PIG_CLASSPATH?

How about avoiding the pig script and just running

java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR  org.apache.pig.Main
<<yourscript>> ?

I grepped through all the code, including the codegenned stuff, and none of
it contains the string PIGDIR (except bin/pig of course).

At this point, assuming none of the above helps (which I have little hope it
will), the best I can say is set up your eclipse environment and step
through pig's main with the debugger.

-D

On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
<[EMAIL PROTECTED]>wrote:

> Hello
>
> This is getting tiresome :)
>
> Today, I tried with a fresh download of pig 0.5 and I get the exact same
> error. Something somewhere just broke down, and it took my machine with it.
> At this point I have no way to run any kind of pig version.
> Needless to say, this is a big problem for me :)
>
> Another thing I tried is to do a fresh build of the 0.6 branch. Same error.
> I even dropped the sym links.
>
> Back to our debugging session:
>
> > svn co
> http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
> A    pig-branch-0.6-take2/test/bin/test-patch.sh
> Checked out external at revision 910901.
>
> Checked out revision 910900.
>
> > cd pig-branch-0.6-take2/
> > ant
> BUILD SUCCESSFUL
> Total time: 26 seconds
>
> > cd bin/
>
> next is to check if all vars are there
> > printenv | grep PIG
> PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
> PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
>
> > printenv | grep HADOOP
> HADOOP_HOME=/home/alex/hadoop/hadoop
> HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> HADOOPDIR=/home/alex/hadoop/hadoop/conf
>
> > pig
> Exception in thread "main" java.lang.NoClassDefFoundError:
> jline/ConsoleReaderInputStream
> Caused by: java.lang.ClassNotFoundException: jline.ConsoleReaderInputStream
>    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> Could not find the main class: org.apache.pig.Main.  Program will exit.
>
> I've sent some email to the list about this problem before. A quick fix is
> to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
> > cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
>
> > pig
> 2010-02-17 11:48:52,843 [main] INFO  org.apache.pig.Main - Logging error
> messages to:
> /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
> 2010-02-17 11:48:53,175 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: file:///
> grunt>
>
> Now at least it will start.
>
> Getting to the initial issue:
>
> the '-f' option:
> > pig -f /home/alex/hadoop/test/test_lite.pig
> 2010-02-17 11:49:28,988 [main] INFO  org.apache.pig.Main - Logging error
> messages to:
> /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
> 2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR 2999:
> Unexpected internal error. Undefined parameter : PIGDIR
> Details at logfile:
> /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
>
> the '- secretDebugCmd' option
>
> pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
> dry run:
> /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
> -Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
> -Dpig.log.file=pig.log
> -Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
> -Dpig.root.logger=INFO,console,DRFA -classpath

Alex,

I just downloaded the latest Pig 0.6 Release Candidate from
http://people.apache.org/~gates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz and
followed the instructions in "Trying the Release" section as outlined in
http://people.apache.org/~gates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt and
without taking any extra step anywhere in between I am able to get Pig
running. You may want to do the same thing. If it works that would be great
else we will be sure that its some environment problem in your setup and we
can take it from there.

Ashutosh

On Wed, Feb 17, 2010 at 13:51, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> Probably unrelated, but I always run from the root directory, not from
> bin/,
> when working with an svn version. Also, never had dev vs core issues. Can
> you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
> pig.jar (not core, not  dev) on $PIG_CLASSPATH?
>
> How about avoiding the pig script and just running
>
> java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR  org.apache.pig.Main
> <<yourscript>> ?
>
> I grepped through all the code, including the codegenned stuff, and none of
> it contains the string PIGDIR (except bin/pig of course).
>
> At this point, assuming none of the above helps (which I have little hope
> it
> will), the best I can say is set up your eclipse environment and step
> through pig's main with the debugger.
>
> -D
>
> On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
> <[EMAIL PROTECTED]>wrote:
>
> > Hello
> >
> > This is getting tiresome :)
> >
> > Today, I tried with a fresh download of pig 0.5 and I get the exact same
> > error. Something somewhere just broke down, and it took my machine with
> it.
> > At this point I have no way to run any kind of pig version.
> > Needless to say, this is a big problem for me :)
> >
> > Another thing I tried is to do a fresh build of the 0.6 branch. Same
> error.
> > I even dropped the sym links.
> >
> > Back to our debugging session:
> >
> > > svn co
> >
> http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
> > A    pig-branch-0.6-take2/test/bin/test-patch.sh
> > Checked out external at revision 910901.
> >
> > Checked out revision 910900.
> >
> > > cd pig-branch-0.6-take2/
> > > ant
> > BUILD SUCCESSFUL
> > Total time: 26 seconds
> >
> > > cd bin/
> >
> > next is to check if all vars are there
> > > printenv | grep PIG
> > PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
> > PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
> >
> > > printenv | grep HADOOP
> > HADOOP_HOME=/home/alex/hadoop/hadoop
> > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> > HADOOPDIR=/home/alex/hadoop/hadoop/conf
> >
> > > pig
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > jline/ConsoleReaderInputStream
> > Caused by: java.lang.ClassNotFoundException:
> jline.ConsoleReaderInputStream
> >    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> >    at java.security.AccessController.doPrivileged(Native Method)
> >    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> >    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> > Could not find the main class: org.apache.pig.Main.  Program will exit.
> >
> > I've sent some email to the list about this problem before. A quick fix
> is
> > to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
> > > cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
> >
> > > pig
> > 2010-02-17 11:48:52,843 [main] INFO  org.apache.pig.Main - Logging error
> > messages to:
> > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
> > 2010-02-17 11:48:53,175 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> Connecting
> > to hadoop file system at: file:///
> > grunt>
> >
> > Now at least it will start.
> >

Hello and sorry for the absence

I went through a hard drive crash, and a 2 day recovery process.

I have incredible news... Pig is working again!!!!!!!!!!

it felt good to finally say it out loud

I've found the problem. Check this out:

--------- This is my test script -----------

-- java -cp $PIGDIR/pig-0.6.1-dev-core.jar:$HADOOPDIR org.apache.pig.Main
test_lite.pig

A = load 't.csv' using PigStorage('\t') as (id: long);
store A into 'output/pig' using PigStorage('\t');

--------------------

I just saved the run instruction in this script to I don't forget it.
Notice the '$PIGDIR' and '$HADOOPDIR'

It apprears this is the source if the evil
  ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
replacing this with
-- $whatever
will give you: ERROR 2999: Unexpected internal error. Undefined parameter :
whatever

Now that it's working again, I'll be back with more issues in no time :)

Thanks a million to everybody, especially Dmitriy!

alex
On Thu, Feb 18, 2010 at 2:50 AM, Ashutosh Chauhan <
[EMAIL PROTECTED]> wrote:

> Alex,
>
> I just downloaded the latest Pig 0.6 Release Candidate from
> http://people.apache.org/~gates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz>and
> followed the instructions in "Trying the Release" section as outlined in
> http://people.apache.org/~gates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt>and
> without taking any extra step anywhere in between I am able to get Pig
> running. You may want to do the same thing. If it works that would be great
> else we will be sure that its some environment problem in your setup and we
> can take it from there.
>
> Ashutosh
>
> On Wed, Feb 17, 2010 at 13:51, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:
>
> > Probably unrelated, but I always run from the root directory, not from
> > bin/,
> > when working with an svn version. Also, never had dev vs core issues. Can
> > you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
> > pig.jar (not core, not  dev) on $PIG_CLASSPATH?
> >
> > How about avoiding the pig script and just running
> >
> > java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR  org.apache.pig.Main
> > <<yourscript>> ?
> >
> > I grepped through all the code, including the codegenned stuff, and none
> of
> > it contains the string PIGDIR (except bin/pig of course).
> >
> > At this point, assuming none of the above helps (which I have little hope
> > it
> > will), the best I can say is set up your eclipse environment and step
> > through pig's main with the debugger.
> >
> > -D
> >
> > On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Hello
> > >
> > > This is getting tiresome :)
> > >
> > > Today, I tried with a fresh download of pig 0.5 and I get the exact
> same
> > > error. Something somewhere just broke down, and it took my machine with
> > it.
> > > At this point I have no way to run any kind of pig version.
> > > Needless to say, this is a big problem for me :)
> > >
> > > Another thing I tried is to do a fresh build of the 0.6 branch. Same
> > error.
> > > I even dropped the sym links.
> > >
> > > Back to our debugging session:
> > >
> > > > svn co
> > >
> >
> http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
> > > A    pig-branch-0.6-take2/test/bin/test-patch.sh
> > > Checked out external at revision 910901.
> > >
> > > Checked out revision 910900.
> > >
> > > > cd pig-branch-0.6-take2/
> > > > ant
> > > BUILD SUCCESSFUL
> > > Total time: 26 seconds
> > >
> > > > cd bin/
> > >
> > > next is to check if all vars are there
> > > > printenv | grep PIG
> > > PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
> > > PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
> > >
> > > > printenv | grep HADOOP
> > > HADOOP_HOME=/home/alex/hadoop/hadoop
> > > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> > > HADOOPDIR=/home/alex/hadoop/hadoop/conf

Still not sure how that's happening, but glad it's working for you now :-).

-D

On Wed, Feb 24, 2010 at 6:15 AM, Alex Parvulescu
<[EMAIL PROTECTED]>wrote:

> Hello and sorry for the absence
>
> I went through a hard drive crash, and a 2 day recovery process.
>
> I have incredible news... Pig is working again!!!!!!!!!!
>
> it felt good to finally say it out loud
>
> I've found the problem. Check this out:
>
> --------- This is my test script -----------
>
> -- java -cp $PIGDIR/pig-0.6.1-dev-core.jar:$HADOOPDIR org.apache.pig.Main
> test_lite.pig
>
> A = load 't.csv' using PigStorage('\t') as (id: long);
> store A into 'output/pig' using PigStorage('\t');
>
> --------------------
>
> I just saved the run instruction in this script to I don't forget it.
> Notice the '$PIGDIR' and '$HADOOPDIR'
>
> It apprears this is the source if the evil
>   ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
> replacing this with
> -- $whatever
> will give you: ERROR 2999: Unexpected internal error. Undefined parameter :
> whatever
>
> Now that it's working again, I'll be back with more issues in no time :)
>
> Thanks a million to everybody, especially Dmitriy!
>
> alex
>
>
> On Thu, Feb 18, 2010 at 2:50 AM, Ashutosh Chauhan <
> [EMAIL PROTECTED]> wrote:
>
> > Alex,
> >
> > I just downloaded the latest Pig 0.6 Release Candidate from
> > http://people.apache.org/~gates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz>
> <http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz
> >and
> > followed the instructions in "Trying the Release" section as outlined in
> > http://people.apache.org/~gates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt>
> <http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt
> >and
> > without taking any extra step anywhere in between I am able to get Pig
> > running. You may want to do the same thing. If it works that would be
> great
> > else we will be sure that its some environment problem in your setup and
> we
> > can take it from there.
> >
> > Ashutosh
> >
> > On Wed, Feb 17, 2010 at 13:51, Dmitriy Ryaboy <[EMAIL PROTECTED]>
> wrote:
> >
> > > Probably unrelated, but I always run from the root directory, not from
> > > bin/,
> > > when working with an svn version. Also, never had dev vs core issues.
> Can
> > > you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
> > > pig.jar (not core, not  dev) on $PIG_CLASSPATH?
> > >
> > > How about avoiding the pig script and just running
> > >
> > > java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR  org.apache.pig.Main
> > > <<yourscript>> ?
> > >
> > > I grepped through all the code, including the codegenned stuff, and
> none
> > of
> > > it contains the string PIGDIR (except bin/pig of course).
> > >
> > > At this point, assuming none of the above helps (which I have little
> hope
> > > it
> > > will), the best I can say is set up your eclipse environment and step
> > > through pig's main with the debugger.
> > >
> > > -D
> > >
> > > On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
> > > <[EMAIL PROTECTED]>wrote:
> > >
> > > > Hello
> > > >
> > > > This is getting tiresome :)
> > > >
> > > > Today, I tried with a fresh download of pig 0.5 and I get the exact
> > same
> > > > error. Something somewhere just broke down, and it took my machine
> with
> > > it.
> > > > At this point I have no way to run any kind of pig version.
> > > > Needless to say, this is a big problem for me :)
> > > >
> > > > Another thing I tried is to do a fresh build of the 0.6 branch. Same
> > > error.
> > > > I even dropped the sym links.
> > > >
> > > > Back to our debugging session:
> > > >
> > > > > svn co
> > > >
> > >
> >
> http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
> > > > A    pig-branch-0.6-take2/test/bin/test-patch.sh
> > > > Checked out external at revision 910901.