Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Pig 0.10 on YARN


Copy link to this message
-
Re: Pig 0.10 on YARN
Konstantin Boudnik 2013-04-09, 06:41
In case you wanna know the actual cause of the issue, please look at
BIGTOP-906. It seems that published Pig artifacts are missing pieces of what's
available in the fat-pig-jar aka pig.jar. And seemingly the only way to deal
with it is to avoid using Maven artifacts and start using the fat one.

I wonder if the artifacts can be fixed somehow...
  Cos

On Fri, Mar 29, 2013 at 03:45PM, Konstantin Boudnik wrote:
> Hey Johnny.
>
> Yup, I've been using pretty much this without -hs artifact. However, adding
> hadoop-mapreduce-client-hs didn't help ;(
>
> Cos
>
> On Fri, Mar 29, 2013 at 03:25PM, Johnny Zhang wrote:
> > Hi, cos:
> > I met the same problem before. Can you try below in your Bigtop pig
> > maven execution .pom ? It will actually resolve other problem as well (for
> > example, you may see Pig cannot get kerberos principle in secure YARN
> > cluster)
> >
> >
> >       <dependencies>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.pig</groupId>
> >
> >           <artifactId>pigsmoke</artifactId>
> >
> >           <version>${pig-smoke.version}</version>
> >
> >           <exclusions>
> >
> >             <exclusion>
> >
> >               <groupId>org.apache.pig</groupId>
> >
> >               <artifactId>pig</artifactId>
> >
> >             </exclusion>
> >
> >           </exclusions>
> >
> >         </dependency>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.hadoop</groupId>
> >
> >           <artifactId>hadoop-common</artifactId>
> >
> >           <version>${hadoop.version}</version>
> >
> >         </dependency>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.hadoop</groupId>
> >
> >           <artifactId>hadoop-hdfs</artifactId>
> >
> >           <version>${hadoop.version}</version>
> >
> >         </dependency>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.hadoop</groupId>
> >
> >           <artifactId>hadoop-mapreduce-client-core</artifactId>
> >
> >           <version>${hadoop.version}</version>
> >
> >         </dependency>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.hadoop</groupId>
> >
> >           <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
> >
> >           <version>${hadoop.version}</version>
> >
> >         </dependency>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.hadoop</groupId>
> >
> >           <artifactId>hadoop-mapreduce-client-hs</artifactId>
> >
> >           <version>${hadoop.version}</version>
> >
> >         </dependency>
> >
> >
> > Johnny
> >
> >
> >
> > On Fri, Mar 29, 2013 at 2:54 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
> >
> > > I certainly agree with you. I have managed to narrow it down to the
> > > difference
> > > between pig-0.10.0-withouthadoop.jar and maven pig-0.10.0.jar.
> > >
> > > Basically, I can run:
> > >
> > > java -cp \
> > > /etc/hadoop/conf:\
> > > /etc/pig/conf:\
> > > /root/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:\
> > >
> > > /root/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:\
> > >
> > > /root/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:\
> > > /root/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar:\
> > > /root/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:\
> > > /usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:\
> > > /root/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-common/2.0.3.1-SNAPSHOT/hadoop-common-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-auth/2.0.3.1-SNAPSHOT/hadoop-auth-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.3.1-SNAPSHOT/hadoop-hdfs-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.0.3.1-SNAPSHOT/hadoop-yarn-api-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.0.3.1-SNAPSHOT/hadoop-yarn-client-2.0.3.1-SNAPSHOT.jar:\