Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> How is pig so much faster than my java MR job?


+
Pavan Sudheendra 2013-09-02, 13:32
+
Dhaval Shah 2013-09-02, 13:45
+
Adrien Mogenet 2013-09-02, 13:50
+
Pavan Sudheendra 2013-09-03, 06:03
Copy link to this message
-
Re: How is pig so much faster than my java MR job?
You are using Scan caching in ur MR java code?   How many mapper and
reducers in case of pig?  How is ur Java MR job written..  a bit more on
its logic pls.

-Anoop-

On Tue, Sep 3, 2013 at 11:33 AM, Pavan Sudheendra <[EMAIL PROTECTED]>wrote:

> Hi all,
> I'm doing a kind of table join across 3 tables in the MR job ( plus doing
> some computation).. It took nearly 19 hours to run with 21 mappers and 21
> reducers.. But with pig it ran in less than 2 hours..
> We are using HBase both as source and sink.. Is this normal?
>
>
> On Mon, Sep 2, 2013 at 7:20 PM, Adrien Mogenet <[EMAIL PROTECTED]
> >wrote:
>
> > You should have a kind of debug/explain mode in Pig, and will show you
> how
> > it does clever things to optimize its excution path.
> >
> >
> > On Mon, Sep 2, 2013 at 3:45 PM, Dhaval Shah <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Java MR code is not optimized/efficiently written while Pig is highly
> > > optimized? Can you give us more details on what exactly you are trying
> to
> > > do and how your Java MR code is written, how many MR jobs for Java vs
> Pig
> > > and so on
> > >
> > > Sent from Yahoo! Mail on Android
> > >
> > >
> >
> >
> > --
> > Adrien Mogenet
> > http://www.borntosegfault.com
> >
>
>
>
> --
> Regards-
> Pavan
>
+
Pavan Sudheendra 2013-09-03, 07:28