Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - LZO & Pig (Elephantbird?)

Copy link to this message
LZO & Pig (Elephantbird?)
Evert Lammerts 2011-01-12, 20:10
Hello list,

I've installed the LZO codecs (https://github.com/kevinweil/hadoop-lzo) and
now I'm looking into using LZO in Pig. Elephant Bird
(https://github.com/kevinweil/elephant-bird) seems to provide some nice
prefab loaders, but it's requirements do not fit out Hadoop installation
(we're on CDH3b2 with Pig 0.7, EB cannot be used with anything > 0.6). Also
the need for Thrift 0.2 is unclear to me - Thrift is now at 0.5.

Now I did find this project, http://code.google.com/p/hadoop-gpl-packing/,
saying EB can handle even Pig 0.8. This confuses me - can I or can I not use
Elephant Bird with Pig 0.7, or even upgrade to Pig 0.8?

Since EB is probably not an option, does anybody have some pointers on how
to use LZO'ed files with Pig?


Evert Lammerts