Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Local vs mapreduce mode


Copy link to this message
-
Local vs mapreduce mode

Dear Pig experts,

I have the following Pig script that works perfectly in local mode. However, in the mapreduce mode I get AU as :

$HADOOP_CONF_DIR fs -cat /scratch/AU/part-m-00000
Warning: $HADOOP_HOME is deprecated.

{}
{}
{}
{}

Both the local mode and the mapreduce mode relation A is set correctly.

Can anyone please tell me what are the recommended ways for debugging the script in mapreduce mode -- logging utilities etc.

REGISTER /users/p529444/software/pig-0.11.1/contrib/piggybank/java/piggybank.jar;
REGISTER /users/p529444/software/pig-0.11.1/parser.jar

DEFINE SequenceFileLoader org.apache.pig.piggybank.storage.SequenceFileLoader();

A = LOAD '/scratch/file.seq' USING SequenceFileLoader AS (key: chararray, value: chararray);
DESCRIBE A;
STORE A into '/scratch/A';

AU
 = FOREACH A GENERATE parser.Parser(key) AS {(id: int, class: chararray,
 name: chararray, begin: int, end: int, probone: chararray, probtwo:
chararray)};
STORE AU into '/scratch/AU';

     
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB