Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> pig 0.9 slower in local mode?


Copy link to this message
-
Re: pig 0.9 slower in local mode?
Cool. Looking forward to trying out 0.9.2+patch.

On Mon, Dec 19, 2011 at 2:31 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> Yes, starting with 0.7 local mode was moved to sit on top of Hadoop's
> local mode (instead of being a completely separate implementation).
> This had two effects, one desirable, one not so much:
>
> 1) get rid of surprises / bugs caused by having to support 2
> completely separate implementations of the runtime
> 2) slow the thing down a lot.
>
> Julien sped up local mode quite a bit in 0.9.2, and we have another
> patch coming that further removes unnecessary slowness, but I'm afraid
> hadoop-based local mode will never be quite as fast as the old
> local-mode...
>
> D
>
> On Mon, Dec 19, 2011 at 2:23 PM, Dexin Wang <[EMAIL PROTECTED]> wrote:
> > I recently switched to pig 0.9.1 and noticed it runs slower than previous
> > version (like 0.6 which was only recent version supported on Amazon
> couple
> > of months ago) in local mode. Haven't tried the timing in hadoop mode
> yet.
> >
> > I figure it is probably due to some extra debugging or some parameter.
> > Anything I can do to make it faster?
> >
> > Thanks,
> > Dexin
>