Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Unable to load data using PigStorage that was previously stored using PigStorage


Copy link to this message
-
Re: Unable to load data using PigStorage that was previously stored using PigStorage
Hi Rusian:

I used PigStorage to store the data that is originally using Pig data type.
It is strange (or a bug in Pig) that I cannot read the data using
PigStorage that have been stored using PigStorage, isn't it?

Best Regards,

Jerry

On Wed, Apr 17, 2013 at 10:52 PM, Ruslan Al-Fakikh <[EMAIL PROTECTED]>wrote:

> The output:
> ({ ([c#11,d#22]),([c#33,d#44]) })
> ()
> looks weird.
>
> Jerry, maybe the problem is in using PigStorage. As its javadoc says:
>
> A load function that parses a line of input into fields using a character
> delimiter
>
> So I guess this is just for simple csv lines.
> But you are trying to load a complicated Map structure as it was formatted
> by previous storing.
> Probably you'll need to write your own Loader for this. Another hint: using
> the -schema paramenter to PigStorage, but I am not sure it can help:(
>
> Ruslan
>
>
> On Wed, Apr 17, 2013 at 11:48 PM, Jerry Lam <[EMAIL PROTECTED]> wrote:
>
> > Hi Rusian:
> >
> > I did a describe B followed by a dump B, the output is:
> > B: {b: {()}}
> >
> > ({ ([c#11,d#22]),([c#33,d#44]) })
> > ()
> >
> > but when I executed
> >
> > C = foreach B generate flatten(b);
> >
> > dump C;
> >
> > I got the exception again...
> >
> > 2013-04-17 15:47:39,933 [Thread-26] WARN
> >  org.apache.hadoop.mapred.LocalJobRunner - job_local_0002
> > java.lang.Exception: java.lang.ClassCastException:
> > org.apache.pig.data.DataByteArray cannot be cast to
> > org.apache.pig.data.DataBag
> > at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:400)
> > Caused by: java.lang.ClassCastException:
> org.apache.pig.data.DataByteArray
> > cannot be cast to org.apache.pig.data.DataBag
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.processInputBag(POProject.java:586)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.getNext(POProject.java:250)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:283)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> > at
> >
> >
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:232)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> > at java.lang.Thread.run(Thread.java:680)
> >
> >
> > Best Regards,
> >
> > Jerry
> >
> >
> > On Wed, Apr 17, 2013 at 3:26 PM, Ruslan Al-Fakikh <[EMAIL PROTECTED]
> > >wrote:
> >
> > > I think that before doing the FLATTEN, you should be 100% sure that
> your
> > > cast worked properly. Can you first DESCRIBE B and then DUMP B right
> > away?
> > > Or probably it just can't be cast in this way. Honestly I don't know
> > > exactly how it works, but here:
> > > http://pig.apache.org/docs/r0.10.0/basic.html#cast
> >