Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> pig 0.9 slower in local mode?


+
Dexin Wang 2011-12-19, 22:23
+
Koji Noguchi 2011-12-19, 22:28
+
Dmitriy Ryaboy 2011-12-19, 22:31
Copy link to this message
-
Re: pig 0.9 slower in local mode?
Cool. Looking forward to trying out 0.9.2+patch.

On Mon, Dec 19, 2011 at 2:31 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> Yes, starting with 0.7 local mode was moved to sit on top of Hadoop's
> local mode (instead of being a completely separate implementation).
> This had two effects, one desirable, one not so much:
>
> 1) get rid of surprises / bugs caused by having to support 2
> completely separate implementations of the runtime
> 2) slow the thing down a lot.
>
> Julien sped up local mode quite a bit in 0.9.2, and we have another
> patch coming that further removes unnecessary slowness, but I'm afraid
> hadoop-based local mode will never be quite as fast as the old
> local-mode...
>
> D
>
> On Mon, Dec 19, 2011 at 2:23 PM, Dexin Wang <[EMAIL PROTECTED]> wrote:
> > I recently switched to pig 0.9.1 and noticed it runs slower than previous
> > version (like 0.6 which was only recent version supported on Amazon
> couple
> > of months ago) in local mode. Haven't tried the timing in hadoop mode
> yet.
> >
> > I figure it is probably due to some extra debugging or some parameter.
> > Anything I can do to make it faster?
> >
> > Thanks,
> > Dexin
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB