Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> problem filtering null values with pig


Copy link to this message
-
Re: problem filtering null values with pig
you are right, it doesn't seam like a null value.
it looks like a chararray. But the expression causes error when comparing a
string with ([longitude#-9.15199849,latitude#38.71179122])

geoinfo_no_nulls = FILTER geoinfo BY $0!='null'

I get
ERROR 2997: Unable to recreate exception from backed error:
org.apache.pig.backend.executionengine.ExecException: ERROR 1071: Cannot
convert a map to a String

what's be the best way to filter only the valid rows, since some of them
are string and others map?

Arian

2012/10/31 Cheolsoo Park <[EMAIL PROTECTED]>

> Hi,
>
> I am not sure what's the problem because I can't reproduce it. To me, null
> values are printed as an empty "( )" not "(null)", so it doesn't seem like
> null.
>
> I am wondering whether OpenJDK is the problem. Can you try Oracle HotSpot
> JDK 1.6 and see that fixes it?
>
> Thanks,
> Cheolsoo
>
> On Wed, Oct 31, 2012 at 1:06 PM, Arian Pasquali <[EMAIL PROTECTED]
> >wrote:
>
> > hey people
> > I'm having some troubles with a silly task, I can´t find a way to filter
> > null values from my rows. This is the result when I dump the object
> > geoinfo:
> >
> > DUMP geoinfo;
> > ([longitude#70.95853,latitude#30.9773])
> > ([longitude#-9.37944507,latitude#38.91780853])
> > (null)
> > (null)
> > (null)
> > ([longitude#-92.64416,latitude#16.73326])
> > (null)
> > (null)
> > ([longitude#-9.15199849,latitude#38.71179122])
> > ([longitude#-9.15210796,latitude#38.71195131])
> >
> > and here is the description
> >
> > DESCRIBE geoinfo;
> > geoinfo: {geoLocation: bytearray}
> >
> > What I'm trying to do is to filter null values like this:
> >
> > geoinfo_no_nulls = FILTER geoinfo BY geoLocation is not null;
> >
> > but the result remains the same. nothing is filtered.
> >
> > I also tried something like this
> >
> > geoinfo_no_nulls = FILTER geoinfo BY geoLocation != 'null';
> >
> >  and I got an error
> >
> > org.apache.pig.backend.executionengine.ExecException: ERROR 1071: Cannot
> > convert a map to a String
> >
> > What am I doing wrong here?
> >
> > env details,
> >
> > Ubuntu 12.04.1 LTS,
> > hadoop-1.0.3
> > pig 0.9.3 version 0.9.3-SNAPSHOT (rexported) compiled Oct 24 2012,
> 19:04:03
> > java version "1.6.0_24" OpenJDK Runtime Environment (IcedTea6 1.11.4)
> > (6b24-1.11.4-1ubuntu0.12.04.1)
> > OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
> >
> >
> > ArianP
> >
>