Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - Pig 0.10 on YARN


+
Konstantin Boudnik 2013-03-29, 06:32
+
Dmitriy Ryaboy 2013-03-29, 16:08
+
Konstantin Boudnik 2013-03-29, 17:38
+
Dmitriy Ryaboy 2013-03-29, 18:52
+
Konstantin Boudnik 2013-03-29, 21:54
+
Johnny Zhang 2013-03-29, 22:25
Copy link to this message
-
Re: Pig 0.10 on YARN
Konstantin Boudnik 2013-03-29, 22:45
Hey Johnny.

Yup, I've been using pretty much this without -hs artifact. However, adding
hadoop-mapreduce-client-hs didn't help ;(

Cos

On Fri, Mar 29, 2013 at 03:25PM, Johnny Zhang wrote:
> Hi, cos:
> I met the same problem before. Can you try below in your Bigtop pig
> maven execution .pom ? It will actually resolve other problem as well (for
> example, you may see Pig cannot get kerberos principle in secure YARN
> cluster)
>
>
>       <dependencies>
>
>         <dependency>
>
>           <groupId>org.apache.pig</groupId>
>
>           <artifactId>pigsmoke</artifactId>
>
>           <version>${pig-smoke.version}</version>
>
>           <exclusions>
>
>             <exclusion>
>
>               <groupId>org.apache.pig</groupId>
>
>               <artifactId>pig</artifactId>
>
>             </exclusion>
>
>           </exclusions>
>
>         </dependency>
>
>         <dependency>
>
>           <groupId>org.apache.hadoop</groupId>
>
>           <artifactId>hadoop-common</artifactId>
>
>           <version>${hadoop.version}</version>
>
>         </dependency>
>
>         <dependency>
>
>           <groupId>org.apache.hadoop</groupId>
>
>           <artifactId>hadoop-hdfs</artifactId>
>
>           <version>${hadoop.version}</version>
>
>         </dependency>
>
>         <dependency>
>
>           <groupId>org.apache.hadoop</groupId>
>
>           <artifactId>hadoop-mapreduce-client-core</artifactId>
>
>           <version>${hadoop.version}</version>
>
>         </dependency>
>
>         <dependency>
>
>           <groupId>org.apache.hadoop</groupId>
>
>           <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
>
>           <version>${hadoop.version}</version>
>
>         </dependency>
>
>         <dependency>
>
>           <groupId>org.apache.hadoop</groupId>
>
>           <artifactId>hadoop-mapreduce-client-hs</artifactId>
>
>           <version>${hadoop.version}</version>
>
>         </dependency>
>
>
> Johnny
>
>
>
> On Fri, Mar 29, 2013 at 2:54 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>
> > I certainly agree with you. I have managed to narrow it down to the
> > difference
> > between pig-0.10.0-withouthadoop.jar and maven pig-0.10.0.jar.
> >
> > Basically, I can run:
> >
> > java -cp \
> > /etc/hadoop/conf:\
> > /etc/pig/conf:\
> > /root/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:\
> >
> > /root/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:\
> >
> > /root/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:\
> > /root/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar:\
> > /root/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:\
> > /usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:\
> > /root/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-common/2.0.3.1-SNAPSHOT/hadoop-common-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-auth/2.0.3.1-SNAPSHOT/hadoop-auth-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.3.1-SNAPSHOT/hadoop-hdfs-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.0.3.1-SNAPSHOT/hadoop-yarn-api-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.0.3.1-SNAPSHOT/hadoop-yarn-client-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.0.3.1-SNAPSHOT/hadoop-yarn-common-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-core-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-common-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-jobclient-2.0.3.1-SNAPSHOT.jar:\
+
Konstantin Boudnik 2013-04-09, 06:41