Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> long parse time


+
Patrick Salami 2013-03-28, 19:51
+
Alan Gates 2013-03-29, 00:15
Copy link to this message
-
Re: long parse time
Thanks for the tip. We are actually using Pig 0.10. I will upgrade to 0.11
and see if that resolves the issue.
On Thu, Mar 28, 2013 at 5:15 PM, Alan Gates <[EMAIL PROTECTED]> wrote:

> What version of Pig are you using?  Unreasonably long parse times were in
> issue in Pig 0.9 and 0.10, I believe those issues were fixed in Pig 0.11.
>
> Alan.
>
> On Mar 28, 2013, at 12:51 PM, Patrick Salami wrote:
>
> > We have some very long pig scripts that run several times per day. We
> > believe that the script parsing process takes very long (about 1h).
> During
> > this time, the pig command just hangs before any output is displayed (I
> am
> > assuming this is the parsing phase). My question is, can this process be
> > optimized by somehow serializing the intermediate parsed script to disk
> > after the parsing phase is complete so that we don't have to go through
> the
> > parsing process each time the script is run (so long as the script itself
> > does not change)? That way, we could then load and run the parsed
> > representation of the script rather than re-parsing it for each run.
> Since
> > this is probably not a readily-available feature, could someone please
> > point me to the right place in the code where this intermediate output
> can
> > be intercepted?
> >
> > Thanks!
>
>
+
Koji Noguchi 2013-04-02, 17:27
+
Koji Noguchi 2013-04-03, 19:40
+
Patrick Salami 2013-04-03, 20:27