Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Problem with a protobuf in EB


Copy link to this message
-
Re: Problem with a protobuf in EB
Hi,

Thanks Dmitriy,  I wasn't aware of these changes, would make life much
easier, given that users do not always have the permissions to add jars to
the $PIG_HOME/lib dirs or classpaths.
In any case I've been planning to make a major update on the
hadoop-gpl-packaging rpms to include all of the latest changes.

Cheers,
 Gerrit

On Mon, May 2, 2011 at 2:27 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> Actually pig does some class loader magic for finding classes in registered
> classes on the front end.
> We recently added that to Elephant bird so that it works when the proto or
> thrift classes aren't already on the classpath and are only registered -- I
> believe I merged that into the 8 branch, so if Gerrit updates his packages
> with the most recent version it should " just work"
>
> D
>
> On Fri, Apr 29, 2011 at 5:11 PM, Gerrit Jansen van Vuuren <
> [EMAIL PROTECTED]> wrote:
>
> > Pig has a backend and front end.
> >  I.e.
> >  Front End:
> >     Pig JVM instance.
> >  Back End
> >     Pig classes running your M/R job on hadoop.
> >
> >  When pig instantiates the same loader in the front and back end to get
> > different information on loading the job files. e.g. Which files to Load?
> > This is decided in the front end, Reading the actual file? This is done
> in
> > the back end.
> >
> >  The java classes for your GPB message needs to be present in the Front
> and
> > Back end.
> >
> >  How?
> >   REGISTER <jar> === Back End
> >   $PIG_HOME/lib/ == Front End
> >
> >
> > Cheers,
> >  Gerrit
> >
> > On Sat, Apr 30, 2011 at 2:02 AM, Kris Coward <[EMAIL PROTECTED]> wrote:
> >
> > >
> > > Here we go:
> > >
> > > META-INF/
> > > META-INF/MANIFEST.MF
> > > com/work/logs/LogFormat$1.class
> > > com/work/logs/LogFormat$Apa$Builder.class
> > > com/work/logs/LogFormat$Apa.class
> > > com/work/logs/LogFormat.class
> > > com/work/logs/LogFormat$Cpu$Builder.class
> > > com/work/logs/LogFormat$Cpu.class
> > > com/work/logs/LogFormat$Evt$Builder.class
> > > com/work/logs/LogFormat$Evt.class
> > > com/work/logs/LogFormat$FirstMsg$Builder.class
> > > com/work/logs/LogFormat$FirstMsg.class
> > > com/work/logs/LogFormat$Gci$Builder.class
> > > com/work/logs/LogFormat$Gci.class
> > > com/work/logs/LogFormat$Inr$Builder.class
> > > com/work/logs/LogFormat$Inr.class
> > > com/work/logs/LogFormat$Ins$Builder.class
> > > com/work/logs/LogFormat$Ins.class
> > > com/work/logs/LogFormat$Mer$Builder.class
> > > com/work/logs/LogFormat$Mer.class
> > > com/work/logs/LogFormat$Mes$Builder.class
> > > com/work/logs/LogFormat$Mes.class
> > > com/work/logs/LogFormat$Mtu$Builder.class
> > > com/work/logs/LogFormat$Mtu.class
> > > com/work/logs/LogFormat$Nei$Builder.class
> > > com/work/logs/LogFormat$Nei.class
> > > com/work/logs/LogFormat$Nes$Builder.class
> > > com/work/logs/LogFormat$Nes.class
> > > com/work/logs/LogFormat$Ntr$Builder.class
> > > com/work/logs/LogFormat$Ntr.class
> > > com/work/logs/LogFormat$Nts$Builder.class
> > > com/work/logs/LogFormat$Nts.class
> > > com/work/logs/LogFormat$Pgr$Builder.class
> > > com/work/logs/LogFormat$Pgr.class
> > > com/work/logs/LogFormat$Psr$Builder.class
> > > com/work/logs/LogFormat$Psr.class
> > > com/work/logs/LogFormat$Pst$Builder.class
> > > com/work/logs/LogFormat$Pst.class
> > > com/work/logs/LogFormat$Ucc$Builder.class
> > > com/work/logs/LogFormat$Ucc.class
> > >
> > > On Fri, Apr 29, 2011 at 04:16:05PM -0700, Dmitriy Ryaboy wrote:
> > > > and the contents of '/home/kris/swineflu/logformats-0.1.2.jar'  (jar
> > -tf)
> > > >
> > > > D
> > > >
> > > > On Fri, Apr 29, 2011 at 1:15 PM, Kris Coward <[EMAIL PROTECTED]> wrote:
> > > >
> > > > >
> > > > > Well I'll send up to the point where it fails and exits, since the
> > rest
> > > > > seems kinda superfluous.. here it is:
> > > > >
> > > > > REGISTER '/usr/local/hadoopgpl/lib/slf4j-api-1.5.8.jar'
> > > > > REGISTER '/usr/local/hadoopgpl/lib/slf4j-log4j12-1.5.10.jar'
> > > > > REGISTER '/usr/local/pig/lib/elephant-bird.jar'
> > > > > REGISTER '/usr/local/pig/lib/hadoop-lzo.jar'
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB