Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Pig 0.10 on YARN


+
Konstantin Boudnik 2013-03-29, 06:32
+
Dmitriy Ryaboy 2013-03-29, 16:08
+
Konstantin Boudnik 2013-03-29, 17:38
+
Dmitriy Ryaboy 2013-03-29, 18:52
+
Konstantin Boudnik 2013-03-29, 21:54
+
Johnny Zhang 2013-03-29, 22:25
+
Konstantin Boudnik 2013-03-29, 22:45
Copy link to this message
-
Re: Pig 0.10 on YARN
In case you wanna know the actual cause of the issue, please look at
BIGTOP-906. It seems that published Pig artifacts are missing pieces of what's
available in the fat-pig-jar aka pig.jar. And seemingly the only way to deal
with it is to avoid using Maven artifacts and start using the fat one.

I wonder if the artifacts can be fixed somehow...
  Cos

On Fri, Mar 29, 2013 at 03:45PM, Konstantin Boudnik wrote:
> Hey Johnny.
>
> Yup, I've been using pretty much this without -hs artifact. However, adding
> hadoop-mapreduce-client-hs didn't help ;(
>
> Cos
>
> On Fri, Mar 29, 2013 at 03:25PM, Johnny Zhang wrote:
> > Hi, cos:
> > I met the same problem before. Can you try below in your Bigtop pig
> > maven execution .pom ? It will actually resolve other problem as well (for
> > example, you may see Pig cannot get kerberos principle in secure YARN
> > cluster)
> >
> >
> >       <dependencies>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.pig</groupId>
> >
> >           <artifactId>pigsmoke</artifactId>
> >
> >           <version>${pig-smoke.version}</version>
> >
> >           <exclusions>
> >
> >             <exclusion>
> >
> >               <groupId>org.apache.pig</groupId>
> >
> >               <artifactId>pig</artifactId>
> >
> >             </exclusion>
> >
> >           </exclusions>
> >
> >         </dependency>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.hadoop</groupId>
> >
> >           <artifactId>hadoop-common</artifactId>
> >
> >           <version>${hadoop.version}</version>
> >
> >         </dependency>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.hadoop</groupId>
> >
> >           <artifactId>hadoop-hdfs</artifactId>
> >
> >           <version>${hadoop.version}</version>
> >
> >         </dependency>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.hadoop</groupId>
> >
> >           <artifactId>hadoop-mapreduce-client-core</artifactId>
> >
> >           <version>${hadoop.version}</version>
> >
> >         </dependency>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.hadoop</groupId>
> >
> >           <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
> >
> >           <version>${hadoop.version}</version>
> >
> >         </dependency>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.hadoop</groupId>
> >
> >           <artifactId>hadoop-mapreduce-client-hs</artifactId>
> >
> >           <version>${hadoop.version}</version>
> >
> >         </dependency>
> >
> >
> > Johnny
> >
> >
> >
> > On Fri, Mar 29, 2013 at 2:54 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
> >
> > > I certainly agree with you. I have managed to narrow it down to the
> > > difference
> > > between pig-0.10.0-withouthadoop.jar and maven pig-0.10.0.jar.
> > >
> > > Basically, I can run:
> > >
> > > java -cp \
> > > /etc/hadoop/conf:\
> > > /etc/pig/conf:\
> > > /root/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:\
> > >
> > > /root/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:\
> > >
> > > /root/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:\
> > > /root/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar:\
> > > /root/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:\
> > > /usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:\
> > > /root/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-common/2.0.3.1-SNAPSHOT/hadoop-common-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-auth/2.0.3.1-SNAPSHOT/hadoop-auth-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.3.1-SNAPSHOT/hadoop-hdfs-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.0.3.1-SNAPSHOT/hadoop-yarn-api-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.0.3.1-SNAPSHOT/hadoop-yarn-client-2.0.3.1-SNAPSHOT.jar:\
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB