Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - How is pig so much faster than my java MR job?


+
Pavan Sudheendra 2013-09-02, 13:32
+
Dhaval Shah 2013-09-02, 13:45
+
Adrien Mogenet 2013-09-02, 13:50
+
Pavan Sudheendra 2013-09-03, 06:03
Copy link to this message
-
Re: How is pig so much faster than my java MR job?
Anoop John 2013-09-03, 07:03
You are using Scan caching in ur MR java code?   How many mapper and
reducers in case of pig?  How is ur Java MR job written..  a bit more on
its logic pls.

-Anoop-

On Tue, Sep 3, 2013 at 11:33 AM, Pavan Sudheendra <[EMAIL PROTECTED]>wrote:

> Hi all,
> I'm doing a kind of table join across 3 tables in the MR job ( plus doing
> some computation).. It took nearly 19 hours to run with 21 mappers and 21
> reducers.. But with pig it ran in less than 2 hours..
> We are using HBase both as source and sink.. Is this normal?
>
>
> On Mon, Sep 2, 2013 at 7:20 PM, Adrien Mogenet <[EMAIL PROTECTED]
> >wrote:
>
> > You should have a kind of debug/explain mode in Pig, and will show you
> how
> > it does clever things to optimize its excution path.
> >
> >
> > On Mon, Sep 2, 2013 at 3:45 PM, Dhaval Shah <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Java MR code is not optimized/efficiently written while Pig is highly
> > > optimized? Can you give us more details on what exactly you are trying
> to
> > > do and how your Java MR code is written, how many MR jobs for Java vs
> Pig
> > > and so on
> > >
> > > Sent from Yahoo! Mail on Android
> > >
> > >
> >
> >
> > --
> > Adrien Mogenet
> > http://www.borntosegfault.com
> >
>
>
>
> --
> Regards-
> Pavan
>
+
Pavan Sudheendra 2013-09-03, 07:28