Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - Pig 0.10 on YARN


+
Konstantin Boudnik 2013-03-29, 06:32
+
Dmitriy Ryaboy 2013-03-29, 16:08
+
Konstantin Boudnik 2013-03-29, 17:38
+
Dmitriy Ryaboy 2013-03-29, 18:52
+
Konstantin Boudnik 2013-03-29, 21:54
Copy link to this message
-
Re: Pig 0.10 on YARN
Johnny Zhang 2013-03-29, 22:25
Hi, cos:
I met the same problem before. Can you try below in your Bigtop pig
maven execution .pom ? It will actually resolve other problem as well (for
example, you may see Pig cannot get kerberos principle in secure YARN
cluster)
      <dependencies>

        <dependency>

          <groupId>org.apache.pig</groupId>

          <artifactId>pigsmoke</artifactId>

          <version>${pig-smoke.version}</version>

          <exclusions>

            <exclusion>

              <groupId>org.apache.pig</groupId>

              <artifactId>pig</artifactId>

            </exclusion>

          </exclusions>

        </dependency>

        <dependency>

          <groupId>org.apache.hadoop</groupId>

          <artifactId>hadoop-common</artifactId>

          <version>${hadoop.version}</version>

        </dependency>

        <dependency>

          <groupId>org.apache.hadoop</groupId>

          <artifactId>hadoop-hdfs</artifactId>

          <version>${hadoop.version}</version>

        </dependency>

        <dependency>

          <groupId>org.apache.hadoop</groupId>

          <artifactId>hadoop-mapreduce-client-core</artifactId>

          <version>${hadoop.version}</version>

        </dependency>

        <dependency>

          <groupId>org.apache.hadoop</groupId>

          <artifactId>hadoop-mapreduce-client-jobclient</artifactId>

          <version>${hadoop.version}</version>

        </dependency>

        <dependency>

          <groupId>org.apache.hadoop</groupId>

          <artifactId>hadoop-mapreduce-client-hs</artifactId>

          <version>${hadoop.version}</version>

        </dependency>
Johnny

On Fri, Mar 29, 2013 at 2:54 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:

> I certainly agree with you. I have managed to narrow it down to the
> difference
> between pig-0.10.0-withouthadoop.jar and maven pig-0.10.0.jar.
>
> Basically, I can run:
>
> java -cp \
> /etc/hadoop/conf:\
> /etc/pig/conf:\
> /root/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:\
>
> /root/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:\
>
> /root/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:\
> /root/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar:\
> /root/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:\
> /usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:\
> /root/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-common/2.0.3.1-SNAPSHOT/hadoop-common-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-auth/2.0.3.1-SNAPSHOT/hadoop-auth-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.3.1-SNAPSHOT/hadoop-hdfs-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.0.3.1-SNAPSHOT/hadoop-yarn-api-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.0.3.1-SNAPSHOT/hadoop-yarn-client-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.0.3.1-SNAPSHOT/hadoop-yarn-common-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-core-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-common-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-jobclient-2.0.3.1-SNAPSHOT.jar:\
> /usr/lib/pig/pig-0.10.0-withouthadoop.jar:\
> /root/.m2/repository/com/google/guava/guava/11.0/guava-11.0.jar:\
> /root/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:\
> /root/.m2/repository/org/antlr/antlr-runtime/3.4/antlr-runtime-3.4.jar:\
>
> /root/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:\
>
> /root/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:\
+
Konstantin Boudnik 2013-03-29, 22:45
+
Konstantin Boudnik 2013-04-09, 06:41