Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - control splits / mapper numbers?


Copy link to this message
-
Re: control splits / mapper numbers?
Yang 2012-01-26, 19:24
found it
SET mapred.min.split.size  12500000;
SET mapred.max.split.size  12500000;
SET pig.noSplitCombination true;

On Wed, Jan 25, 2012 at 6:06 PM, Yang <[EMAIL PROTECTED]> wrote:

> this question was asked a few days before, after that, I dug around the
> source code and found some knobs to turn,
>
> but after adding the following, I still could not get pig to use smaller
> split sizes
>
> any ideas?
>
> -- this does not work for pig
> -- SET mapred.map.tasks 100;
> SET pig.mapsplits.count 100;
> SET mapred.max.jobs.per.node 1;
> --- following changed after 0.21
> --- SET mapred.min.split.size  100;
> SET mapreduce.input.fileinputformat.split.minsize 100;
> SET mapreduce.input.fileinputformat.split.maxsize 100;
> SET pig.splitCombination false;
> SET pig.noSplitCombination true;
> SET pig.maxCombinedSplitSize 100;
>
>