Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Pig job is taking more time than Java M/R


+
praveenesh kumar 2012-01-16, 05:47
+
Prashant Kommireddi 2012-01-16, 06:12
+
praveenesh kumar 2012-01-16, 06:59
+
Prashant Kommireddi 2012-01-16, 07:13
Copy link to this message
-
Re: Pig job is taking more time than Java M/R
Using PARALLEL I can reduce the total number of reducers and can reduce the
execution time.

Thanks,
Praveenesh

On Mon, Jan 16, 2012 at 12:43 PM, Prashant Kommireddi
<[EMAIL PROTECTED]>wrote:

> That explains it. Pig computes # of reducers using a heuristic based on
> input dataset size (1 reducer per GB). You would always want to use
> PARALLEL if the data being forwarded to reducers is not a lot.
>
> Please take a look at (PARALLEL syntax) :
> http://pig.apache.org/docs/r0.9.1/basic.html#distinct
>
> Thanks,
> Prashant
>
> On Sun, Jan 15, 2012 at 10:59 PM, praveenesh kumar <[EMAIL PROTECTED]
> >wrote:
>
> > I am using Apache Pig version 0.11.0-SNAPSHOT (r1225753) build from trunk
> > and Hadoop 0.20.205
> > Nothing else was running that time on cluster that time. and there was no
> > waiting for map-reduce slots.
> > Only difference I saw was for my Java M/R job, only 40 reducers were
> > running
> > whereas my pig job was running 457 reducers. I guess it may be because of
> > so many reducers running.
> > Can I control number of reducers running ?
> >
> > Thanks,
> > Praveenesh
> >
> >
> > On Mon, Jan 16, 2012 at 11:42 AM, Prashant Kommireddi
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Hi Praveenesh,
> > >
> > > You can use 'EXPLAIN' to understand what Pig is doing under the hood
> (MR
> > > plan)
> > > http://pig.apache.org/docs/r0.9.1/test.html#explain
> > >
> > > What version of Pig and Hadoop are you using? I have never seen such a
> > huge
> > > difference between Java MR and Pig. At the time you ran Pig, was the
> > > cluster idle or did you have other jobs running at the same time? Did
> you
> > > make sure the job was not waiting on Map or Reduce slots being made
> > > available?
> > >
> > > Thanks,
> > > Prashant
> > >
> > > On Sun, Jan 15, 2012 at 9:47 PM, praveenesh kumar <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Hey Guys,
> > > >
> > > > Is there anyway through which I can see the M/R jobs that pig runs
> > > > internally for a given pig script ?
> > > > I wanted to get unique values for a particular column.
> > > >
> > > > For that I wrote the following script:
> > > >
> > > > Data = Load 'Data.csv' using PigStorage(',');
> > > > IDs = FOREACH Data GENERATE $0;
> > > > UniqueID = Distinct IDs;
> > > > Dump UniqueID;
> > > >
> > > > Is it the write/best way to get unique values of a particular column
> ?
> > > >
> > > > The reason why I am asking is, I ran the above script on my cluster,
> it
> > > > took around 30 minutes to finish.
> > > > However, for the same thing, when I wrote traditional java M/R code,
> it
> > > > took only 10 minutes.
> > > >
> > > > So I want to see what Pig is doing internally.
> > > > Can anyone tell what could be the reason for such behaviour ? How
> can I
> > > > decrease Pig Execution time ?
> > > >
> > > > Thanks,
> > > > Praveenesh
> > > >
> > >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB