Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> UDF casting failures on nested maps/objects.


Copy link to this message
-
UDF casting failures on nested maps/objects.
Hi,

I've got some objects originally loaded, using the JSON loader from
elephantbird, into nested maps, and subsequently stored using
LZOPigStorage after various stages of processing.

When I subsequently load these nested maps, and pass them into a UDF for
modification, pretty much any attempt to reference nested objects
produces ClassCastExceptions where org.apache.pig.data.DataByteArray
cannot be cast to whatever it is that I'm trying to get at.

That all said, if I tear apart the map in pig latin, and pass all its
bits and pieces as arguments to the UDF for reassembly, these exceptions
don't happen, so this is what I've been doing, but it gets pretty ugly
and complicated at times (especially when the layers of nesting involve
arrays that I have to FLATTEN out and then group back together).

Looking at the stored data, I've noticed that nested maps are serialized
differently than the top-level map, so I've developed a suspicion that
there's some extra bit of magic that Pig is using to parse/cast these
sub-maps that I just don't know about (to be able to call it in my
UDFs). If such a beast exists, and someone could provide me with a
pointer, I'd appreciate it.

Also, the cluster I'm working on right now is using version 0.8.1-cdh3u4
of Pig. I poked arond JIRA to see if this issue has been previously
observed (and if I should bug the relevant ops-folk that much harder to
upgrade our pig), but found nothing. If this has somehow been fixed in a
later version (or perhaps if someone can recommend a storage class that
doesn't cause this problem in the first place), that pointer would also
be very much appreciated.

Thanks,
Kris

--
Kris Coward http://unripe.melon.org/
GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB 12B3
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB