Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - Pig 0.10 on YARN


+
Konstantin Boudnik 2013-03-29, 06:32
+
Dmitriy Ryaboy 2013-03-29, 16:08
+
Konstantin Boudnik 2013-03-29, 17:38
+
Dmitriy Ryaboy 2013-03-29, 18:52
Copy link to this message
-
Re: Pig 0.10 on YARN
Konstantin Boudnik 2013-03-29, 21:54
I certainly agree with you. I have managed to narrow it down to the difference
between pig-0.10.0-withouthadoop.jar and maven pig-0.10.0.jar.

Basically, I can run:

java -cp \
/etc/hadoop/conf:\
/etc/pig/conf:\
/root/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:\
/root/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:\
/root/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:\
/root/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar:\
/root/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:\
/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:\
/root/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-common/2.0.3.1-SNAPSHOT/hadoop-common-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-auth/2.0.3.1-SNAPSHOT/hadoop-auth-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.3.1-SNAPSHOT/hadoop-hdfs-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.0.3.1-SNAPSHOT/hadoop-yarn-api-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.0.3.1-SNAPSHOT/hadoop-yarn-client-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.0.3.1-SNAPSHOT/hadoop-yarn-common-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-core-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-common-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-jobclient-2.0.3.1-SNAPSHOT.jar:\
/usr/lib/pig/pig-0.10.0-withouthadoop.jar:\
/root/.m2/repository/com/google/guava/guava/11.0/guava-11.0.jar:\
/root/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:\
/root/.m2/repository/org/antlr/antlr-runtime/3.4/antlr-runtime-3.4.jar:\
/root/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:\
/root/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:\
/root/.m2/repository/org/antlr/stringtemplate/3.2.1/stringtemplate-3.2.1.jar:\
/usr/lib/pig/lib/automaton.jar:\
build/ivy/lib/Pig/junit-4.5.jar:\
build/pig-0.10.0-SNAPSHOT-smoketests.jar:\
/root/.m2/repository/org/apache/pig/pigunit/0.10.0-SNAPSHOT/pigunit-0.10.0-SNAPSHOT.jar
\
-Dpigunit.exectype.cluster=true \
org.junit.runner.JUnitCore org.apache.pig.test.pigunit.TestPigTest

But once
  /usr/lib/pig/pig-0.10.0-withouthadoop.jar
is replaced with
  /root/.m2/repository/org/apache/pig/pig/0.10.0/pig-0.10.0.jar
all hell breaks loose and submission doesn't happen anymore ;( And as before -
no error messages nor exceptions appear anywhere on the surface.

Cos

On Fri, Mar 29, 2013 at 11:52AM, Dmitriy Ryaboy wrote:
> I missed the part where you are running it from JUnit. Still suspect a
> classpath issue :). Can you dump your classpath as seen by JUnit, and what
> happens when you run "-secretDebugCmd" and see if something is missing?
>
> -D
>
>
> On Fri, Mar 29, 2013 at 10:38 AM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>
> > There's no problem with Pig itself. If I use Pig CLI I can run some
> > PigLatin
> > scripts and they go all the way to the completion.
> >
> > The problem arises while running TestPigTest _programmatically_ e.g. using
> > JUnit runner. Is there any secret debug options in this case? I've looked
> > through the code but didn't spot anything immediately.
> >
> > I will give "-secretDebugCmd" a try anyway - it might give me some ideas...
> >
> > Appreciate the help!
> >   Cos
> >
> > On Fri, Mar 29, 2013 at 09:08AM, Dmitriy Ryaboy wrote:
> > > Can you run `pig -secretDebugCmd` and look at your paths? I suspect
> > there's
> > > something going on there, like not having the right configs or hadoop
> > jars.
> > >
> > >
+
Johnny Zhang 2013-03-29, 22:25
+
Konstantin Boudnik 2013-03-29, 22:45
+
Konstantin Boudnik 2013-04-09, 06:41