Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Making Pig run faster in local mode


+
Malcolm Tye 2013-01-04, 17:35
+
Jonathan Coveney 2013-01-04, 19:07
+
Russell Jurney 2013-01-04, 22:04
Copy link to this message
-
RE: Making Pig run faster in local mode
Hi,
  It's Pig 0.10.0. Here's some timings I took. I have more than 3
files to process, but I just started out with 3 files to get some numbers.

# Files Time(s)
1 28
2 48
3 73
Cheolsoo, the documentation does seem to indicate that you will only get 1
reducer when running in local mode, and I've tested this out using the
parallel statement on the group by's to verify that is the case. When you
say to use MR mode, do you mean install hadoop onto the node ?
Thanks

Malc

-----Original Message-----
From: Russell Jurney [mailto:[EMAIL PROTECTED]]
Sent: 04 January 2013 22:05
To: [EMAIL PROTECTED]
Subject: Re: Making Pig run faster in local mode

+1 wasn't there a slowdown bug a little while ago?

What version of Pig?
On Jan 4, 2013 11:07 AM, "Jonathan Coveney" <[EMAIL PROTECTED]> wrote:

> How long is it taking?
>
>
> 2013/1/4 Malcolm Tye <[EMAIL PROTECTED]>
>
> > Hi,
> >
> >                 Any ideas on how to make Pig run quicker when running it
> in
> > local mode ?
> >
> >
> >
> > I'm processing 3 files of about 13MB each with 3 group by statements in
> my
> > script which seem to suck up the time. There's no joins
> >
> >
> >
> > Increasing the heap size has made no difference and it doesn't use all
> that
> > anyway.
> >
> >
> >
> > I'm on default settings apart from that.
> >
> >
> >
> >
> >
> > Thanks
> >
> >
> >
> > Malc
> >
> >
>
+
Cheolsoo Park 2013-01-07, 19:55
+
Cheolsoo Park 2013-01-07, 19:56
+
Dmitriy Ryaboy 2013-01-08, 07:36
+
Malcolm Tye 2013-01-21, 14:01
+
Cheolsoo Park 2013-01-04, 19:07
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB