Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> SUM


Strange, that looks right to me. What happens if you try the 'AS'
statement anyhow?

--jacob
@thedatachef

On Sun, 2011-04-24 at 18:22 +0200, pob wrote:
> Hello,
>
> pom = foreach rows generate myUDF.toTuple($1); -- reading data
> describe pom
> pom: {y: {t: (domain: chararray,spam: int,size: long,time: float)}}
>
> data = foreach pom generate flatten($0);
> grunt> describe data;
> data: {y::domain: chararray,y::spam: int,y::size: long,y::time: float}
>
>
> I thing they are casted fine, right?
>
> UDF is python one with decorator
> @outputSchema("y:bag{t:tuple(domain:chararray, spam:int, size:long,
> time:float)}")
>
> Thanks
>
>
>
> 2011/4/24 Jacob Perkins <[EMAIL PROTECTED]>
>
> > You're getting a 'ClassCastException' because the contents of the bags
> > are DataByteArray and not long (or cannot be cast to long). I suspect
> > that you're generating the contents of the bag in some way from a UDF,
> > no?
> >
> > You need to either declare the output schema explicitly in the UDF or
> > just use the 'AS' statement. For example, say you have a UDF that sums
> > two numbers:
> >
> > data   = LOAD 'foobar' AS (int:a, int:b);
> > summed = FOREACH data GENERATE MyFancySummingUDF(a,b) AS (sum:int);
> > DUMP summed;
> >
> > --jacob
> > @thedatachef
> >
> > On Sun, 2011-04-24 at 18:02 +0200, pob wrote:
> > > x = foreach g2 generate group, data.(size);
> > > dump x;
> > >
> > > ((drm,0),{(464868)})
> > > ((drm,1),{(464868)})
> > > ((snezz,0),{(8073),(8073)})
> > >
> > > but:
> > > x = foreach g2 generate group, SUM(data.size);
> > >
> > >
> > >
> > >
> > > 2011-04-24 18:02:18,910 [Thread-793] WARN
> > >  org.apache.hadoop.mapred.LocalJobRunner - job_local_0038
> > > org.apache.pig.backend.executionengine.ExecException: ERROR 2106: Error
> > > while computing sum in Initial
> > > at org.apache.pig.builtin.LongSum$Initial.exec(LongSum.java:87)
> > > at org.apache.pig.builtin.LongSum$Initial.exec(LongSum.java:65)
> > > at
> > >
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:229)
> > > at
> > >
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:273)
> > > at
> > >
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:343)
> > > at
> > >
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:291)
> > > at
> > >
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:276)
> > > at
> > >
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:256)
> > > at
> > >
> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:236)
> > > at
> > >
> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
> > > at
> > >
> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
> > > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at
> > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> > > Caused by: java.lang.ClassCastException:
> > org.apache.pig.data.DataByteArray
> > > cannot be cast to java.lang.Long
> > > at org.apache.pig.builtin.LongSum$Initial.exec(LongSum.java:79)
> > > ... 14 more
> > > 2011-04-24 18:02:19,213 [main] INFO
> > >
> >  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > > - HadoopJobId: job_local_0038
> > > 2011-04-24 18:02:19,213 [main] INFO
> > >
> >  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher