Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - a simple logic causes very long compiling time on pig 0.10.0


Copy link to this message
-
Re: a simple logic causes very long compiling time on pig 0.10.0
Jonathan Coveney 2012-06-26, 23:20
Thanks!

2012/6/26 Danfeng Li <[EMAIL PROTECTED]>

> Ticket PIG-2769 is created for this one
> https://issues.apache.org/jira/browse/PIG-2769
>
> Dan
>
>
> -----Original Message-----
> From: Jonathan Coveney [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, June 26, 2012 4:02 PM
> To: [EMAIL PROTECTED]
> Subject: Re: a simple logic causes very long compiling time on pig 0.10.0
>
> This is a great find. Please file a ticket.
>
> My guess is that there is some backtracking in the parser, which explodes
> for large values.
>
> 2012/6/26 Clay B. <[EMAIL PROTECTED]>
>
> > It's worth pointing out that Pig 0.9.2 also runs quickly; we only see
> > the degradation with Pig 0.10.0.
> >
> > The degradation in performance seems to have a knee as 4 or 5
> > conditionals works as expected but as presented, the script takes
> > about 6 minutes at the
> > GRUNT> prompt after hitting enter; before any Hadoop execution.
> >
> > -Clay
> >
> >
> > On Tue, 26 Jun 2012, Danfeng Li wrote:
> >
> >
> >> We found the following simple logic will cause very long compiling
> >> time for pig 0.10.0, while using pig 0.8.1, everything is fine.
> >>
> >>
> >>
> >> A = load 'A.txt' using PigStorage()  AS (m: int);
> >>
> >>
> >>
> >> B = FOREACH A {
> >>
> >>     days_str = (chararray)
> >>
> >>         (m == 1 ? 31:
> >>
> >>         (m == 2 ? 28:
> >>
> >>         (m == 3 ? 31:
> >>
> >>         (m == 4 ? 30:
> >>
> >>         (m == 5 ? 31:
> >>
> >>         (m == 6 ? 30:
> >>
> >>         (m == 7 ? 31:
> >>
> >>         (m == 8 ? 31:
> >>
> >>         (m == 9 ? 30:
> >>
> >>         (m == 10 ? 31:
> >>
> >>         (m == 11 ? 30:31)))))))))));
> >>
> >> GENERATE
> >>
> >>    days_str as days_str;
> >>
> >> }
> >>
> >> store B into 'B';
> >>
> >>
> >>
> >> here’s the pig version we used in the test
> >>
> >> Apache Pig version 0.10.0-SNAPSHOT (rexported)
> >>
> >>
> >>
> >>
> >>
> >> Attached is the pig code and an example input file.
> >>
> >>
> >>
> >> Dan
> >>
> >>
> >>
> >>
> >>
> >>
> >>
>