Hi folks,
I have a pig script that right now looks like this:

…
likes = FILTER main_set BY blah == 'a' AND meh == 'b';
likes_time = FOREACH likes GENERATE date, 'likes' AS type;

dislikes = FILTER main_set BY blah == 'b' AND meh == 'c';
dislikes_time = FOREACH dislikes GENERATE date, 'dislikes' AS type;

newuserregs = FILTER main_set BY blah == 'c' AND meh == 'd';
newuserregs_time = FOREACH dislikes GENERATE date, 'newuserregs' as type;
...

all_time = UNION likes_time, dislikes_time, newuserregs_time;
…

As you can see, what I'm doing is filtering the main_set repeatedly and generating based on that, and then unioning everything back together. This means a lot of extra map jobs, which is a lot of extra work. Really, thinking about it in terms of mapping, I should be able to do things in one run. Any idea what the pig syntax would be for that? Is there something like a GENERATE conditional, where I could do something like:

all_time = FOREACH main_set GENERATE date, 'likes' IF (blah == 'a' AND meh == 'b')
 'dislikes' IF (blah == 'b' AND meh == 'c')
 'dislikes' IF (blah == 'c' AND meh == 'd') AS type;

Running this in just one map job would be very awesome and would speed this script up a ton, I'm thinking. Ideas? Advice?

Eli

Sorry, forgot to mention: I know I can use a UDF for this, but was wondering if/hoping there's a pure PIG approach.

Eli

On Oct 23, 2012, at 11:51 PM, Eli Finkelshteyn wrote:

> Hi folks,
> I have a pig script that right now looks like this:
>
> …
> likes = FILTER main_set BY blah == 'a' AND meh == 'b';
> likes_time = FOREACH likes GENERATE date, 'likes' AS type;
>
> dislikes = FILTER main_set BY blah == 'b' AND meh == 'c';
> dislikes_time = FOREACH dislikes GENERATE date, 'dislikes' AS type;
>
> newuserregs = FILTER main_set BY blah == 'c' AND meh == 'd';
> newuserregs_time = FOREACH dislikes GENERATE date, 'newuserregs' as type;
> ...
>
> all_time = UNION likes_time, dislikes_time, newuserregs_time;
> …
>
> As you can see, what I'm doing is filtering the main_set repeatedly and generating based on that, and then unioning everything back together. This means a lot of extra map jobs, which is a lot of extra work. Really, thinking about it in terms of mapping, I should be able to do things in one run. Any idea what the pig syntax would be for that? Is there something like a GENERATE conditional, where I could do something like:
>
> all_time = FOREACH main_set GENERATE date, 'likes' IF (blah == 'a' AND meh == 'b')
>  'dislikes' IF (blah == 'b' AND meh == 'c')
>  'dislikes' IF (blah == 'c' AND meh == 'd') AS type;
>
> Running this in just one map job would be very awesome and would speed this script up a ton, I'm thinking. Ideas? Advice?
>
> Eli

Try
http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#SPLIT

On Oct 24, 2012, at 2:51 AM, Eli Finkelshteyn <[EMAIL PROTECTED]> wrote:

> Hi folks,
> I have a pig script that right now looks like this:
>
> …
> likes = FILTER main_set BY blah == 'a' AND meh == 'b';
> likes_time = FOREACH likes GENERATE date, 'likes' AS type;
>
> dislikes = FILTER main_set BY blah == 'b' AND meh == 'c';
> dislikes_time = FOREACH dislikes GENERATE date, 'dislikes' AS type;
>
> newuserregs = FILTER main_set BY blah == 'c' AND meh == 'd';
> newuserregs_time = FOREACH dislikes GENERATE date, 'newuserregs' as type;
> ...
>
> all_time = UNION likes_time, dislikes_time, newuserregs_time;
> …
>
> As you can see, what I'm doing is filtering the main_set repeatedly and generating based on that, and then unioning everything back together. This means a lot of extra map jobs, which is a lot of extra work. Really, thinking about it in terms of mapping, I should be able to do things in one run. Any idea what the pig syntax would be for that? Is there something like a GENERATE conditional, where I could do something like:
>
> all_time = FOREACH main_set GENERATE date, 'likes' IF (blah == 'a' AND meh == 'b')
>  'dislikes' IF (blah == 'b' AND meh == 'c')
>  'dislikes' IF (blah == 'c' AND meh == 'd') AS type;
>
> Running this in just one map job would be very awesome and would speed this script up a ton, I'm thinking. Ideas? Advice?
>
> Eli

Are you sure Pig is spawning extra map jobs for this?  The multi-query optimizer should be pushing these back together into one job.

If it isn't, you should be able to accomplish the same thing with trinary logic and a single filter:

all = foreach main_set ((blah == 'a' and meh == 'b') ? 'likes' : ((blah == 'b' and meh == 'c') ? 'disklikes' : ((blah == 'c' and meh =='d') ? 'newuserregs' : ''))) as type;
all_time = filter all by type != '';

(Not sure about all the parenthesis placement, as I didn't run it.)

Alan.

On Oct 24, 2012, at 2:51 AM, Eli Finkelshteyn wrote:

> Hi folks,
> I have a pig script that right now looks like this:
>
> …
> likes = FILTER main_set BY blah == 'a' AND meh == 'b';
> likes_time = FOREACH likes GENERATE date, 'likes' AS type;
>
> dislikes = FILTER main_set BY blah == 'b' AND meh == 'c';
> dislikes_time = FOREACH dislikes GENERATE date, 'dislikes' AS type;
>
> newuserregs = FILTER main_set BY blah == 'c' AND meh == 'd';
> newuserregs_time = FOREACH dislikes GENERATE date, 'newuserregs' as type;
> ...
>
> all_time = UNION likes_time, dislikes_time, newuserregs_time;
> …
>
> As you can see, what I'm doing is filtering the main_set repeatedly and generating based on that, and then unioning everything back together. This means a lot of extra map jobs, which is a lot of extra work. Really, thinking about it in terms of mapping, I should be able to do things in one run. Any idea what the pig syntax would be for that? Is there something like a GENERATE conditional, where I could do something like:
>
> all_time = FOREACH main_set GENERATE date, 'likes' IF (blah == 'a' AND meh == 'b')
>  'dislikes' IF (blah == 'b' AND meh == 'c')
>  'dislikes' IF (blah == 'c' AND meh == 'd') AS type;
>
> Running this in just one map job would be very awesome and would speed this script up a ton, I'm thinking. Ideas? Advice?
>
> Eli