Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - More on issue with local vs mapreduce mode


Copy link to this message
-
More on issue with local vs mapreduce mode
Sameer Tilak 2013-11-05, 23:45
Hello,

My script in the local mode works perfectly. The same script does not work in the mapreduce mode. For the local mode, the o/p is saved in the current directory, where as for the mapreduce mode I use /scrach directory on HDFS.

Local mode:

A = LOAD 'file.seq' USING SequenceFileLoader AS (key: chararray, value: chararray);
DESCRIBE A;
STORE A into 'A';

AU = FOREACH A GENERATE FLATTEN(parser.customFilter(key,'AAAAA'));
STORE AU into 'AU';    
Mapreduce mode:

A = LOAD '/scratch/file.seq' USING SequenceFileLoader AS (key: chararray, value: chararray);
DESCRIBE A;
STORE A into '/scratch/A';

AU = FOREACH A GENERATE FLATTEN(parser.customFilter(key,'AAAAA'));
STORE AU into '/scratch/AU';

Can someone please point me to tools that I can use to debug the script in mapreduce mode? Also, any thoughts on why this might be happening would be great!