Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Load Pig metadata from file?


Copy link to this message
-
Load Pig metadata from file?

Here is a sample LOAD statement from Programming Pig book:

daily = load 'NYSE_daily' as (exchange:chararray, symbol:chararray,
            date:chararray, open:float, high:float, low:float, close:float,
            volume:int, adj_close:float);

In my case, there are around 250 columns to load. So, I created a file, say, metadata.dat with its contents as follows:

 (exchange:chararray, symbol:chararray,

            date:chararray, open:float, high:float, low:float, close:float,

            volume:int, adj_close:float)

My load statement now looks like

daily = load 'NYSE_daily' as $md;

and the execution looks like.

pig -f script.pig -param md=$(cat metadata.dat)

However, I get the following error in this method:

ERROR 1000: Error during parsing. Lexical error at line 9, column 0.  Encountered: <EOF> after : ""

Copying the contents of the file in appropriate place works fine. But the pig script is cluttered with the metdata and I would like to separate it from the script. Any ideas?

HCatLoader() does not seem to be available on my system.