Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Pig 0.10 on YARN


+
Konstantin Boudnik 2013-03-29, 06:32
+
Dmitriy Ryaboy 2013-03-29, 16:08
+
Konstantin Boudnik 2013-03-29, 17:38
+
Dmitriy Ryaboy 2013-03-29, 18:52
+
Konstantin Boudnik 2013-03-29, 21:54
+
Johnny Zhang 2013-03-29, 22:25
Copy link to this message
-
Re: Pig 0.10 on YARN
Hey Johnny.

Yup, I've been using pretty much this without -hs artifact. However, adding
hadoop-mapreduce-client-hs didn't help ;(

Cos

On Fri, Mar 29, 2013 at 03:25PM, Johnny Zhang wrote:
> Hi, cos:
> I met the same problem before. Can you try below in your Bigtop pig
> maven execution .pom ? It will actually resolve other problem as well (for
> example, you may see Pig cannot get kerberos principle in secure YARN
> cluster)
>
>
>       <dependencies>
>
>         <dependency>
>
>           <groupId>org.apache.pig</groupId>
>
>           <artifactId>pigsmoke</artifactId>
>
>           <version>${pig-smoke.version}</version>
>
>           <exclusions>
>
>             <exclusion>
>
>               <groupId>org.apache.pig</groupId>
>
>               <artifactId>pig</artifactId>
>
>             </exclusion>
>
>           </exclusions>
>
>         </dependency>
>
>         <dependency>
>
>           <groupId>org.apache.hadoop</groupId>
>
>           <artifactId>hadoop-common</artifactId>
>
>           <version>${hadoop.version}</version>
>
>         </dependency>
>
>         <dependency>
>
>           <groupId>org.apache.hadoop</groupId>
>
>           <artifactId>hadoop-hdfs</artifactId>
>
>           <version>${hadoop.version}</version>
>
>         </dependency>
>
>         <dependency>
>
>           <groupId>org.apache.hadoop</groupId>
>
>           <artifactId>hadoop-mapreduce-client-core</artifactId>
>
>           <version>${hadoop.version}</version>
>
>         </dependency>
>
>         <dependency>
>
>           <groupId>org.apache.hadoop</groupId>
>
>           <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
>
>           <version>${hadoop.version}</version>
>
>         </dependency>
>
>         <dependency>
>
>           <groupId>org.apache.hadoop</groupId>
>
>           <artifactId>hadoop-mapreduce-client-hs</artifactId>
>
>           <version>${hadoop.version}</version>
>
>         </dependency>
>
>
> Johnny
>
>
>
> On Fri, Mar 29, 2013 at 2:54 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>
> > I certainly agree with you. I have managed to narrow it down to the
> > difference
> > between pig-0.10.0-withouthadoop.jar and maven pig-0.10.0.jar.
> >
> > Basically, I can run:
> >
> > java -cp \
> > /etc/hadoop/conf:\
> > /etc/pig/conf:\
> > /root/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:\
> >
> > /root/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:\
> >
> > /root/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:\
> > /root/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar:\
> > /root/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:\
> > /usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:\
> > /root/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-common/2.0.3.1-SNAPSHOT/hadoop-common-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-auth/2.0.3.1-SNAPSHOT/hadoop-auth-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.3.1-SNAPSHOT/hadoop-hdfs-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.0.3.1-SNAPSHOT/hadoop-yarn-api-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.0.3.1-SNAPSHOT/hadoop-yarn-client-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.0.3.1-SNAPSHOT/hadoop-yarn-common-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-core-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-common-2.0.3.1-SNAPSHOT.jar:\
> >
> > /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-jobclient-2.0.3.1-SNAPSHOT.jar:\
+
Konstantin Boudnik 2013-04-09, 06:41
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB