Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Problem with a protobuf in EB


Copy link to this message
-
Re: Problem with a protobuf in EB
Gerrit Jansen van Vuuren 2011-05-02, 08:43
Hi,

Thanks Dmitriy,  I wasn't aware of these changes, would make life much
easier, given that users do not always have the permissions to add jars to
the $PIG_HOME/lib dirs or classpaths.
In any case I've been planning to make a major update on the
hadoop-gpl-packaging rpms to include all of the latest changes.

Cheers,
 Gerrit

On Mon, May 2, 2011 at 2:27 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> Actually pig does some class loader magic for finding classes in registered
> classes on the front end.
> We recently added that to Elephant bird so that it works when the proto or
> thrift classes aren't already on the classpath and are only registered -- I
> believe I merged that into the 8 branch, so if Gerrit updates his packages
> with the most recent version it should " just work"
>
> D
>
> On Fri, Apr 29, 2011 at 5:11 PM, Gerrit Jansen van Vuuren <
> [EMAIL PROTECTED]> wrote:
>
> > Pig has a backend and front end.
> >  I.e.
> >  Front End:
> >     Pig JVM instance.
> >  Back End
> >     Pig classes running your M/R job on hadoop.
> >
> >  When pig instantiates the same loader in the front and back end to get
> > different information on loading the job files. e.g. Which files to Load?
> > This is decided in the front end, Reading the actual file? This is done
> in
> > the back end.
> >
> >  The java classes for your GPB message needs to be present in the Front
> and
> > Back end.
> >
> >  How?
> >   REGISTER <jar> === Back End
> >   $PIG_HOME/lib/ == Front End
> >
> >
> > Cheers,
> >  Gerrit
> >
> > On Sat, Apr 30, 2011 at 2:02 AM, Kris Coward <[EMAIL PROTECTED]> wrote:
> >
> > >
> > > Here we go:
> > >
> > > META-INF/
> > > META-INF/MANIFEST.MF
> > > com/work/logs/LogFormat$1.class
> > > com/work/logs/LogFormat$Apa$Builder.class
> > > com/work/logs/LogFormat$Apa.class
> > > com/work/logs/LogFormat.class
> > > com/work/logs/LogFormat$Cpu$Builder.class
> > > com/work/logs/LogFormat$Cpu.class
> > > com/work/logs/LogFormat$Evt$Builder.class
> > > com/work/logs/LogFormat$Evt.class
> > > com/work/logs/LogFormat$FirstMsg$Builder.class
> > > com/work/logs/LogFormat$FirstMsg.class
> > > com/work/logs/LogFormat$Gci$Builder.class
> > > com/work/logs/LogFormat$Gci.class
> > > com/work/logs/LogFormat$Inr$Builder.class
> > > com/work/logs/LogFormat$Inr.class
> > > com/work/logs/LogFormat$Ins$Builder.class
> > > com/work/logs/LogFormat$Ins.class
> > > com/work/logs/LogFormat$Mer$Builder.class
> > > com/work/logs/LogFormat$Mer.class
> > > com/work/logs/LogFormat$Mes$Builder.class
> > > com/work/logs/LogFormat$Mes.class
> > > com/work/logs/LogFormat$Mtu$Builder.class
> > > com/work/logs/LogFormat$Mtu.class
> > > com/work/logs/LogFormat$Nei$Builder.class
> > > com/work/logs/LogFormat$Nei.class
> > > com/work/logs/LogFormat$Nes$Builder.class
> > > com/work/logs/LogFormat$Nes.class
> > > com/work/logs/LogFormat$Ntr$Builder.class
> > > com/work/logs/LogFormat$Ntr.class
> > > com/work/logs/LogFormat$Nts$Builder.class
> > > com/work/logs/LogFormat$Nts.class
> > > com/work/logs/LogFormat$Pgr$Builder.class
> > > com/work/logs/LogFormat$Pgr.class
> > > com/work/logs/LogFormat$Psr$Builder.class
> > > com/work/logs/LogFormat$Psr.class
> > > com/work/logs/LogFormat$Pst$Builder.class
> > > com/work/logs/LogFormat$Pst.class
> > > com/work/logs/LogFormat$Ucc$Builder.class
> > > com/work/logs/LogFormat$Ucc.class
> > >
> > > On Fri, Apr 29, 2011 at 04:16:05PM -0700, Dmitriy Ryaboy wrote:
> > > > and the contents of '/home/kris/swineflu/logformats-0.1.2.jar'  (jar
> > -tf)
> > > >
> > > > D
> > > >
> > > > On Fri, Apr 29, 2011 at 1:15 PM, Kris Coward <[EMAIL PROTECTED]> wrote:
> > > >
> > > > >
> > > > > Well I'll send up to the point where it fails and exits, since the
> > rest
> > > > > seems kinda superfluous.. here it is:
> > > > >
> > > > > REGISTER '/usr/local/hadoopgpl/lib/slf4j-api-1.5.8.jar'
> > > > > REGISTER '/usr/local/hadoopgpl/lib/slf4j-log4j12-1.5.10.jar'
> > > > > REGISTER '/usr/local/pig/lib/elephant-bird.jar'
> > > > > REGISTER '/usr/local/pig/lib/hadoop-lzo.jar'