Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Pig 0.10 on YARN


+
Konstantin Boudnik 2013-03-29, 06:32
+
Dmitriy Ryaboy 2013-03-29, 16:08
+
Konstantin Boudnik 2013-03-29, 17:38
+
Dmitriy Ryaboy 2013-03-29, 18:52
Copy link to this message
-
Re: Pig 0.10 on YARN
I certainly agree with you. I have managed to narrow it down to the difference
between pig-0.10.0-withouthadoop.jar and maven pig-0.10.0.jar.

Basically, I can run:

java -cp \
/etc/hadoop/conf:\
/etc/pig/conf:\
/root/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:\
/root/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:\
/root/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:\
/root/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar:\
/root/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:\
/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:\
/root/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-common/2.0.3.1-SNAPSHOT/hadoop-common-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-auth/2.0.3.1-SNAPSHOT/hadoop-auth-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.3.1-SNAPSHOT/hadoop-hdfs-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.0.3.1-SNAPSHOT/hadoop-yarn-api-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.0.3.1-SNAPSHOT/hadoop-yarn-client-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.0.3.1-SNAPSHOT/hadoop-yarn-common-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-core-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-common-2.0.3.1-SNAPSHOT.jar:\
/root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-jobclient-2.0.3.1-SNAPSHOT.jar:\
/usr/lib/pig/pig-0.10.0-withouthadoop.jar:\
/root/.m2/repository/com/google/guava/guava/11.0/guava-11.0.jar:\
/root/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:\
/root/.m2/repository/org/antlr/antlr-runtime/3.4/antlr-runtime-3.4.jar:\
/root/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:\
/root/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:\
/root/.m2/repository/org/antlr/stringtemplate/3.2.1/stringtemplate-3.2.1.jar:\
/usr/lib/pig/lib/automaton.jar:\
build/ivy/lib/Pig/junit-4.5.jar:\
build/pig-0.10.0-SNAPSHOT-smoketests.jar:\
/root/.m2/repository/org/apache/pig/pigunit/0.10.0-SNAPSHOT/pigunit-0.10.0-SNAPSHOT.jar
\
-Dpigunit.exectype.cluster=true \
org.junit.runner.JUnitCore org.apache.pig.test.pigunit.TestPigTest

But once
  /usr/lib/pig/pig-0.10.0-withouthadoop.jar
is replaced with
  /root/.m2/repository/org/apache/pig/pig/0.10.0/pig-0.10.0.jar
all hell breaks loose and submission doesn't happen anymore ;( And as before -
no error messages nor exceptions appear anywhere on the surface.

Cos

On Fri, Mar 29, 2013 at 11:52AM, Dmitriy Ryaboy wrote:
> I missed the part where you are running it from JUnit. Still suspect a
> classpath issue :). Can you dump your classpath as seen by JUnit, and what
> happens when you run "-secretDebugCmd" and see if something is missing?
>
> -D
>
>
> On Fri, Mar 29, 2013 at 10:38 AM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>
> > There's no problem with Pig itself. If I use Pig CLI I can run some
> > PigLatin
> > scripts and they go all the way to the completion.
> >
> > The problem arises while running TestPigTest _programmatically_ e.g. using
> > JUnit runner. Is there any secret debug options in this case? I've looked
> > through the code but didn't spot anything immediately.
> >
> > I will give "-secretDebugCmd" a try anyway - it might give me some ideas...
> >
> > Appreciate the help!
> >   Cos
> >
> > On Fri, Mar 29, 2013 at 09:08AM, Dmitriy Ryaboy wrote:
> > > Can you run `pig -secretDebugCmd` and look at your paths? I suspect
> > there's
> > > something going on there, like not having the right configs or hadoop
> > jars.
> > >
> > >
+
Johnny Zhang 2013-03-29, 22:25
+
Konstantin Boudnik 2013-03-29, 22:45
+
Konstantin Boudnik 2013-04-09, 06:41
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB