Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> No of reducers


Copy link to this message
-
Re: No of reducers
In pig 0.8 the default number of reducers changed from 1 to a value computed based on input data size -
http://pig.apache.org/docs/r0.8.1/cookbook.html#Use+the+Parallel+Features

-Thejas
On 5/27/11 6:46 AM, "Jonathan Coveney" <[EMAIL PROTECTED]> wrote:

SET default_parallel X; will set the PARALLEL keyword for all parallel
functions (ie set the reducers for the job)

I am not sure how the default is calculated...for a while it was set to 1 I
believe, ostensibly to force people to set it to something more reasonable.

2011/5/27 Harsh J <[EMAIL PROTECTED]>

> The PARALLEL keyword controls the number of reducers used in the job.
> If unspecified, a default number is applied. Is this what you're
> looking for?
>
> On Fri, May 27, 2011 at 3:46 PM, Sudharsan Sampath
> <[EMAIL PROTECTED]> wrote:
> >
> > Hi,
> >
> > Is there a reference on how the number of reducers required for a job is
> calculated?
> >
> > Thanks
> > Sudharsan S
> >
> >
>
>
>
> --
> Harsh J
>

--

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB