Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Problem with a protobuf in EB


Copy link to this message
-
Re: Problem with a protobuf in EB
Pig has a backend and front end.
 I.e.
  Front End:
     Pig JVM instance.
  Back End
     Pig classes running your M/R job on hadoop.

 When pig instantiates the same loader in the front and back end to get
different information on loading the job files. e.g. Which files to Load?
This is decided in the front end, Reading the actual file? This is done in
the back end.

 The java classes for your GPB message needs to be present in the Front and
Back end.

 How?
   REGISTER <jar> === Back End
   $PIG_HOME/lib/ == Front End
Cheers,
 Gerrit

On Sat, Apr 30, 2011 at 2:02 AM, Kris Coward <[EMAIL PROTECTED]> wrote:

>
> Here we go:
>
> META-INF/
> META-INF/MANIFEST.MF
> com/work/logs/LogFormat$1.class
> com/work/logs/LogFormat$Apa$Builder.class
> com/work/logs/LogFormat$Apa.class
> com/work/logs/LogFormat.class
> com/work/logs/LogFormat$Cpu$Builder.class
> com/work/logs/LogFormat$Cpu.class
> com/work/logs/LogFormat$Evt$Builder.class
> com/work/logs/LogFormat$Evt.class
> com/work/logs/LogFormat$FirstMsg$Builder.class
> com/work/logs/LogFormat$FirstMsg.class
> com/work/logs/LogFormat$Gci$Builder.class
> com/work/logs/LogFormat$Gci.class
> com/work/logs/LogFormat$Inr$Builder.class
> com/work/logs/LogFormat$Inr.class
> com/work/logs/LogFormat$Ins$Builder.class
> com/work/logs/LogFormat$Ins.class
> com/work/logs/LogFormat$Mer$Builder.class
> com/work/logs/LogFormat$Mer.class
> com/work/logs/LogFormat$Mes$Builder.class
> com/work/logs/LogFormat$Mes.class
> com/work/logs/LogFormat$Mtu$Builder.class
> com/work/logs/LogFormat$Mtu.class
> com/work/logs/LogFormat$Nei$Builder.class
> com/work/logs/LogFormat$Nei.class
> com/work/logs/LogFormat$Nes$Builder.class
> com/work/logs/LogFormat$Nes.class
> com/work/logs/LogFormat$Ntr$Builder.class
> com/work/logs/LogFormat$Ntr.class
> com/work/logs/LogFormat$Nts$Builder.class
> com/work/logs/LogFormat$Nts.class
> com/work/logs/LogFormat$Pgr$Builder.class
> com/work/logs/LogFormat$Pgr.class
> com/work/logs/LogFormat$Psr$Builder.class
> com/work/logs/LogFormat$Psr.class
> com/work/logs/LogFormat$Pst$Builder.class
> com/work/logs/LogFormat$Pst.class
> com/work/logs/LogFormat$Ucc$Builder.class
> com/work/logs/LogFormat$Ucc.class
>
> On Fri, Apr 29, 2011 at 04:16:05PM -0700, Dmitriy Ryaboy wrote:
> > and the contents of '/home/kris/swineflu/logformats-0.1.2.jar'  (jar -tf)
> >
> > D
> >
> > On Fri, Apr 29, 2011 at 1:15 PM, Kris Coward <[EMAIL PROTECTED]> wrote:
> >
> > >
> > > Well I'll send up to the point where it fails and exits, since the rest
> > > seems kinda superfluous.. here it is:
> > >
> > > REGISTER '/usr/local/hadoopgpl/lib/slf4j-api-1.5.8.jar'
> > > REGISTER '/usr/local/hadoopgpl/lib/slf4j-log4j12-1.5.10.jar'
> > > REGISTER '/usr/local/pig/lib/elephant-bird.jar'
> > > REGISTER '/usr/local/pig/lib/hadoop-lzo.jar'
> > > REGISTER '/usr/local/pig/lib/piggybank.jar'
> > > REGISTER '/usr/local/pig/lib/jackson-core-asl-1.0.1.jar'
> > > REGISTER '/usr/local/pig/lib/jackson-mapper-asl-1.0.1.jar'
> > > REGISTER '/usr/local/pig/lib/jsp-2.1-6.1.4.jar'
> > > REGISTER '/home/kris/swineflu/com.kontagent.swineflu.jar'
> > > REGISTER '/home/kris/swineflu/logformats-0.1.2.jar'
> > >
> > > %declare storage
> > > com.twitter.elephantbird.pig.proto.LzoProtobuffB64LinePigStore
> > > %declare loader
> > > com.twitter.elephantbird.pig.proto.LzoProtobuffB64LinePigStore
> > >
> > > -- load the raw data from HDFS
> > > apaNew = LOAD '$infile/apa' USING $loader('apa');
> > > apaTable = LOAD '$firstfile/apa' USING $loader('firstp');
> > >
> > >
> > > (where $infile and $firstfile are passed as parameters at runtime, and
> > > the files were tested as existing)
> > >
> > > Cheers,
> > > Kris
> > >
> > > On Fri, Apr 29, 2011 at 01:00:55PM -0700, Dmitriy Ryaboy wrote:
> > > > Odd.. can you send the full pig script including the register
> statements?
> > > >
> > > > On Fri, Apr 29, 2011 at 11:38 AM, Kris Coward <[EMAIL PROTECTED]>
> wrote:
> > > >
> > > > >
> > > > > So I've recently added a protocol/schema to a collection I got from
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB