Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Load Pig metadata from file?


Copy link to this message
-
Load Pig metadata from file?

Here is a sample LOAD statement from Programming Pig book:

daily = load 'NYSE_daily' as (exchange:chararray, symbol:chararray,
            date:chararray, open:float, high:float, low:float, close:float,
            volume:int, adj_close:float);

In my case, there are around 250 columns to load. So, I created a file, say, metadata.dat with its contents as follows:

 (exchange:chararray, symbol:chararray,

            date:chararray, open:float, high:float, low:float, close:float,

            volume:int, adj_close:float)

My load statement now looks like

daily = load 'NYSE_daily' as $md;

and the execution looks like.

pig -f script.pig -param md=$(cat metadata.dat)

However, I get the following error in this method:

ERROR 1000: Error during parsing. Lexical error at line 9, column 0.  Encountered: <EOF> after : ""

Copying the contents of the file in appropriate place works fine. But the pig script is cluttered with the metdata and I would like to separate it from the script. Any ideas?

HCatLoader() does not seem to be available on my system.

     
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB