Pig, mail # user - RE: this nested foreach worked in 0.8.1, no longer works in 0.10

this nested foreach worked in 0.8.1, no longer works in 0.10

Steve Bernstein 2012-05-31, 19:09

We upgraded from Pig 0.8.1 to 0.10 and the following nested foreach no longer works:

actionBagGrouped = GROUP actionBag BY (deal_id,month);
dealCounts = FOREACH actionBagGrouped {
   sent = FILTER actionBag BY action=='sent';
   opened = FILTER actionBag BY action=='opened';
   clicked = FILTER actionBag BY action=='clicked';
   purchased = FILTER actionBag BY action=='purchased';

   GENERATE
   FLATTEN(group),
   COUNT(sent) AS sentCount,
   COUNT(opened) AS openedCount,
   COUNT(clicked) AS clickedCount,
   COUNT(purchased) AS purchasedCount;
};
I get:
"Invalid scalar projection: action : A column needs to be projected from a relation for it to be used as a scalar"
Banging my head against a wall-the problem is with the nest op sent = Filter...

Help!
Thanks-
Steve

Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Jonathan Coveney 2012-05-31, 20:08

Can you "describe actionBagGrouped" and paste it?

2012/5/31 Steve Bernstein <[EMAIL PROTECTED]>

> We upgraded from Pig 0.8.1 to 0.10 and the following nested foreach no
> longer works:
>
> actionBagGrouped = GROUP actionBag BY (deal_id,month);
> dealCounts = FOREACH actionBagGrouped {
> sent = FILTER actionBag BY action=='sent';
> opened = FILTER actionBag BY action=='opened';
> clicked = FILTER actionBag BY action=='clicked';
> purchased = FILTER actionBag BY action=='purchased';
>
> GENERATE
> FLATTEN(group),
> COUNT(sent) AS sentCount,
> COUNT(opened) AS openedCount,
> COUNT(clicked) AS clickedCount,
> COUNT(purchased) AS purchasedCount;
> };
> I get:
> "Invalid scalar projection: action : A column needs to be projected from a
> relation for it to be used as a scalar"
> Banging my head against a wall-the problem is with the nest op sent > Filter...
>
> Help!
> Thanks-
> Steve
>

RE: this nested foreach worked in 0.8.1, no longer works in 0.10

Steve Bernstein 2012-05-31, 20:24

Yes, it's:
actionBagGrouped: {group: (deal_id: chararray,month: chararray),actionBag: {userID: chararray,month: chararray,deal_id: chararray,action: chararray}}

_____________
Steve Bernstein
VP, Analytics
Rearden Commerce, Inc.

+1.408.499.0961 Mobile

deem.com | reardencommerce.com
-----Original Message-----
From: Jonathan Coveney [mailto:[EMAIL PROTECTED]]
Sent: Thursday, May 31, 2012 1:09 PM
To: [EMAIL PROTECTED]
Subject: Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Can you "describe actionBagGrouped" and paste it?

2012/5/31 Steve Bernstein <[EMAIL PROTECTED]>

> We upgraded from Pig 0.8.1 to 0.10 and the following nested foreach no
> longer works:
>
> actionBagGrouped = GROUP actionBag BY (deal_id,month); dealCounts =
> FOREACH actionBagGrouped {
> sent = FILTER actionBag BY action=='sent';
> opened = FILTER actionBag BY action=='opened';
> clicked = FILTER actionBag BY action=='clicked';
> purchased = FILTER actionBag BY action=='purchased';
>
> GENERATE
> FLATTEN(group),
> COUNT(sent) AS sentCount,
> COUNT(opened) AS openedCount,
> COUNT(clicked) AS clickedCount,
> COUNT(purchased) AS purchasedCount; };
> I get:
> "Invalid scalar projection: action : A column needs to be projected
> from a relation for it to be used as a scalar"
> Banging my head against a wall-the problem is with the nest op sent =
> Filter...
>
> Help!
> Thanks-
> Steve
>

Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Dmitriy Ryaboy 2012-06-01, 09:02

Do you have relations ned sent, opened, etc, elsewhere in the script?

On May 31, 2012, at 12:09 PM, Steve Bernstein <[EMAIL PROTECTED]> wrote:

> We upgraded from Pig 0.8.1 to 0.10 and the following nested foreach no longer works:
>
> actionBagGrouped = GROUP actionBag BY (deal_id,month);
> dealCounts = FOREACH actionBagGrouped {
> sent = FILTER actionBag BY action=='sent';
> opened = FILTER actionBag BY action=='opened';
> clicked = FILTER actionBag BY action=='clicked';
> purchased = FILTER actionBag BY action=='purchased';
>
> GENERATE
> FLATTEN(group),
> COUNT(sent) AS sentCount,
> COUNT(opened) AS openedCount,
> COUNT(clicked) AS clickedCount,
> COUNT(purchased) AS purchasedCount;
> };
> I get:
> "Invalid scalar projection: action : A column needs to be projected from a relation for it to be used as a scalar"
> Banging my head against a wall-the problem is with the nest op sent = Filter...
>
> Help!
> Thanks-
> Steve

RE: this nested foreach worked in 0.8.1, no longer works in 0.10

Steve Bernstein 2012-06-01, 17:25

No, all those relations embedded in the nested foreach are unique to it, not used elsewhere.
SB

-----Original Message-----
From: Dmitriy Ryaboy [mailto:[EMAIL PROTECTED]]
Sent: Friday, June 01, 2012 2:03 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Do you have relations ned sent, opened, etc, elsewhere in the script?

On May 31, 2012, at 12:09 PM, Steve Bernstein <[EMAIL PROTECTED]> wrote:

> We upgraded from Pig 0.8.1 to 0.10 and the following nested foreach no longer works:
>
> actionBagGrouped = GROUP actionBag BY (deal_id,month); dealCounts =
> FOREACH actionBagGrouped {
> sent = FILTER actionBag BY action=='sent';
> opened = FILTER actionBag BY action=='opened';
> clicked = FILTER actionBag BY action=='clicked';
> purchased = FILTER actionBag BY action=='purchased';
>
> GENERATE
> FLATTEN(group),
> COUNT(sent) AS sentCount,
> COUNT(opened) AS openedCount,
> COUNT(clicked) AS clickedCount,
> COUNT(purchased) AS purchasedCount; };
> I get:
> "Invalid scalar projection: action : A column needs to be projected from a relation for it to be used as a scalar"
> Banging my head against a wall-the problem is with the nest op sent = Filter...
>
> Help!
> Thanks-
> Steve

Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Subir S 2012-06-02, 11:04

Can you try sent = FILTER actionBag BY actionBag.action=='sent';

Although not sure why it does not work. I also use pig 0.8.1 and similar
constructs work for me.

On Fri, Jun 1, 2012 at 10:55 PM, Steve Bernstein
<[EMAIL PROTECTED]>wrote:

> No, all those relations embedded in the nested foreach are unique to it,
> not used elsewhere.
> SB
>
> -----Original Message-----
> From: Dmitriy Ryaboy [mailto:[EMAIL PROTECTED]]
> Sent: Friday, June 01, 2012 2:03 AM
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: this nested foreach worked in 0.8.1, no longer works in 0.10
>
> Do you have relations ned sent, opened, etc, elsewhere in the script?
>
> On May 31, 2012, at 12:09 PM, Steve Bernstein <[EMAIL PROTECTED]>
> wrote:
>
> > We upgraded from Pig 0.8.1 to 0.10 and the following nested foreach no
> longer works:
> >
> > actionBagGrouped = GROUP actionBag BY (deal_id,month); dealCounts > > FOREACH actionBagGrouped {
> > sent = FILTER actionBag BY action=='sent';
> > opened = FILTER actionBag BY action=='opened';
> > clicked = FILTER actionBag BY action=='clicked';
> > purchased = FILTER actionBag BY action=='purchased';
> >
> > GENERATE
> > FLATTEN(group),
> > COUNT(sent) AS sentCount,
> > COUNT(opened) AS openedCount,
> > COUNT(clicked) AS clickedCount,
> > COUNT(purchased) AS purchasedCount; };
> > I get:
> > "Invalid scalar projection: action : A column needs to be projected from
> a relation for it to be used as a scalar"
> > Banging my head against a wall-the problem is with the nest op sent > Filter...
> >
> > Help!
> > Thanks-
> > Steve
>

RE: this nested foreach worked in 0.8.1, no longer works in 0.10

Steve Bernstein 2012-06-03, 16:24

Since this seems to do with project, before I wrote to the list, I'd already tried actionBag.sent and actionBagGrouped::actionBag.sent. No joy. Is this a bug? Seems like others would have stumbled across it already.
SB
________________________________________
From: Subir S [[EMAIL PROTECTED]]
Sent: Saturday, June 02, 2012 4:04 AM
To: [EMAIL PROTECTED]
Subject: Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Can you try sent = FILTER actionBag BY actionBag.action=='sent';

Although not sure why it does not work. I also use pig 0.8.1 and similar
constructs work for me.

On Fri, Jun 1, 2012 at 10:55 PM, Steve Bernstein
<[EMAIL PROTECTED]>wrote:

> No, all those relations embedded in the nested foreach are unique to it,
> not used elsewhere.
> SB
>
> -----Original Message-----
> From: Dmitriy Ryaboy [mailto:[EMAIL PROTECTED]]
> Sent: Friday, June 01, 2012 2:03 AM
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: this nested foreach worked in 0.8.1, no longer works in 0.10
>
> Do you have relations ned sent, opened, etc, elsewhere in the script?
>
> On May 31, 2012, at 12:09 PM, Steve Bernstein <[EMAIL PROTECTED]>
> wrote:
>
> > We upgraded from Pig 0.8.1 to 0.10 and the following nested foreach no
> longer works:
> >
> > actionBagGrouped = GROUP actionBag BY (deal_id,month); dealCounts =
> > FOREACH actionBagGrouped {
> > sent = FILTER actionBag BY action=='sent';
> > opened = FILTER actionBag BY action=='opened';
> > clicked = FILTER actionBag BY action=='clicked';
> > purchased = FILTER actionBag BY action=='purchased';
> >
> > GENERATE
> > FLATTEN(group),
> > COUNT(sent) AS sentCount,
> > COUNT(opened) AS openedCount,
> > COUNT(clicked) AS clickedCount,
> > COUNT(purchased) AS purchasedCount; };
> > I get:
> > "Invalid scalar projection: action : A column needs to be projected from
> a relation for it to be used as a scalar"
> > Banging my head against a wall-the problem is with the nest op sent =
> Filter...
> >
> > Help!
> > Thanks-
> > Steve
>

Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Jagat 2012-06-03, 18:08

Hello everyone ,

I also came across one bug in 0.10.0 would like to confirm following
behaviour which i observed.

myGroup = group myData by ( SUBSTRING(NAME,0,1) as FIRST_LETTER;

Assume NAME is present in myData

Syntax error, unexpected symbol at or near 'SUBSTRING'

The above code used to work in 0.8 version , with 0.10 version it gives
error above.

Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Subir S 2012-06-05, 12:53

It was very nice to know answers to these, since it will help while
upgrading from 0.8.1 to 0.10.0 pig.

On Sun, Jun 3, 2012 at 11:38 PM, Jagat <[EMAIL PROTECTED]> wrote:

> Hello everyone ,
>
> I also came across one bug in 0.10.0 would like to confirm following
> behaviour which i observed.
>
> myGroup = group myData by ( SUBSTRING(NAME,0,1) as FIRST_LETTER;
>
> Assume NAME is present in myData
>
> Syntax error, unexpected symbol at or near 'SUBSTRING'
>
> The above code used to work in 0.8 version , with 0.10 version it gives
> error above.
>

Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Dmitriy Ryaboy 2012-06-06, 02:33

Steve,
Scalar projection errors usually happen when you use "foo.bar" syntax
instead of "foo::bar" (and a relation called foo happens to exist) or,
possibly, if there is a relation you create inside a nested foreach,
and another one with identical name outside. The former doesn't seem
to be present in the snippet you provided. The latter has been fixed
iirc (and I think it actually manifested in a different error, don't
remember off the top of my head). Would really need to see the
complete script to reproduce the error and see what's going on.

Jagat -- are you sure the code you pasted is what you are getting
errors on? For starters, it doesn't have balanced parenthesis. It's
also unclear to me what you are trying to do with the "as" clause.

D

On Tue, Jun 5, 2012 at 5:53 AM, Subir S <[EMAIL PROTECTED]> wrote:
> It was very nice to know answers to these, since it will help while
> upgrading from 0.8.1 to 0.10.0 pig.
>
> On Sun, Jun 3, 2012 at 11:38 PM, Jagat <[EMAIL PROTECTED]> wrote:
>
>> Hello everyone ,
>>
>> I also came across one bug in 0.10.0 would like to confirm following
>> behaviour which i observed.
>>
>> myGroup = group myData by ( SUBSTRING(NAME,0,1) as FIRST_LETTER;
>>
>> Assume NAME is present in myData
>>
>> Syntax error, unexpected symbol at or near 'SUBSTRING'
>>
>> The above code used to work in 0.8 version , with 0.10 version it gives
>> error above.
>>

Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Jagat 2012-06-06, 07:41

Hello Dmitriy,

My apology for missing braces , actually i removed extra stuff from the
code to make it general.

as i am using just to give alias name.

You can read it as SUBSTRING inside group By clause for 0.10

On Wed, Jun 6, 2012 at 8:03 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> Steve,
> Scalar projection errors usually happen when you use "foo.bar" syntax
> instead of "foo::bar" (and a relation called foo happens to exist) or,
> possibly, if there is a relation you create inside a nested foreach,
> and another one with identical name outside. The former doesn't seem
> to be present in the snippet you provided. The latter has been fixed
> iirc (and I think it actually manifested in a different error, don't
> remember off the top of my head). Would really need to see the
> complete script to reproduce the error and see what's going on.
>
> Jagat -- are you sure the code you pasted is what you are getting
> errors on? For starters, it doesn't have balanced parenthesis. It's
> also unclear to me what you are trying to do with the "as" clause.
>
> D
>
> On Tue, Jun 5, 2012 at 5:53 AM, Subir S <[EMAIL PROTECTED]> wrote:
> > It was very nice to know answers to these, since it will help while
> > upgrading from 0.8.1 to 0.10.0 pig.
> >
> > On Sun, Jun 3, 2012 at 11:38 PM, Jagat <[EMAIL PROTECTED]> wrote:
> >
> >> Hello everyone ,
> >>
> >> I also came across one bug in 0.10.0 would like to confirm following
> >> behaviour which i observed.
> >>
> >> myGroup = group myData by ( SUBSTRING(NAME,0,1) as FIRST_LETTER;
> >>
> >> Assume NAME is present in myData
> >>
> >> Syntax error, unexpected symbol at or near 'SUBSTRING'
> >>
> >> The above code used to work in 0.8 version , with 0.10 version it gives
> >> error above.
> >>
>

RE: this nested foreach worked in 0.8.1, no longer works in 0.10

Steve Bernstein 2012-06-06, 18:44

Meanwhile, I'm moving forward running this script on 0.8.1 with no issues. Probably going to back off on using 0.10 for now and wait for a more stable release.

-----Original Message-----
From: Steve Bernstein [mailto:[EMAIL PROTECTED]]
Sent: Sunday, June 03, 2012 9:25 AM
To: [EMAIL PROTECTED]
Subject: RE: this nested foreach worked in 0.8.1, no longer works in 0.10

Since this seems to do with project, before I wrote to the list, I'd already tried actionBag.sent and actionBagGrouped::actionBag.sent. No joy. Is this a bug? Seems like others would have stumbled across it already.
SB
________________________________________
From: Subir S [[EMAIL PROTECTED]]
Sent: Saturday, June 02, 2012 4:04 AM
To: [EMAIL PROTECTED]
Subject: Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Can you try sent = FILTER actionBag BY actionBag.action=='sent';

Although not sure why it does not work. I also use pig 0.8.1 and similar constructs work for me.

On Fri, Jun 1, 2012 at 10:55 PM, Steve Bernstein
<[EMAIL PROTECTED]>wrote:

> No, all those relations embedded in the nested foreach are unique to
> it, not used elsewhere.
> SB
>
> -----Original Message-----
> From: Dmitriy Ryaboy [mailto:[EMAIL PROTECTED]]
> Sent: Friday, June 01, 2012 2:03 AM
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: this nested foreach worked in 0.8.1, no longer works in
> 0.10
>
> Do you have relations ned sent, opened, etc, elsewhere in the script?
>
> On May 31, 2012, at 12:09 PM, Steve Bernstein
> <[EMAIL PROTECTED]>
> wrote:
>
> > We upgraded from Pig 0.8.1 to 0.10 and the following nested foreach
> > no
> longer works:
> >
> > actionBagGrouped = GROUP actionBag BY (deal_id,month); dealCounts =
> > FOREACH actionBagGrouped {
> > sent = FILTER actionBag BY action=='sent';
> > opened = FILTER actionBag BY action=='opened';
> > clicked = FILTER actionBag BY action=='clicked';
> > purchased = FILTER actionBag BY action=='purchased';
> >
> > GENERATE
> > FLATTEN(group),
> > COUNT(sent) AS sentCount,
> > COUNT(opened) AS openedCount,
> > COUNT(clicked) AS clickedCount,
> > COUNT(purchased) AS purchasedCount;
> > }; I get:
> > "Invalid scalar projection: action : A column needs to be projected
> > from
> a relation for it to be used as a scalar"
> > Banging my head against a wall-the problem is with the nest op sent
> > > Filter...
> >
> > Help!
> > Thanks-
> > Steve
>

Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Dmitriy Ryaboy 2012-06-06, 18:55

Any chance you can send the full script?

On Jun 6, 2012, at 11:44 AM, Steve Bernstein <[EMAIL PROTECTED]> wrote:

> Meanwhile, I'm moving forward running this script on 0.8.1 with no issues. Probably going to back off on using 0.10 for now and wait for a more stable release.
>
> -----Original Message-----
> From: Steve Bernstein [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, June 03, 2012 9:25 AM
> To: [EMAIL PROTECTED]
> Subject: RE: this nested foreach worked in 0.8.1, no longer works in 0.10
>
> Since this seems to do with project, before I wrote to the list, I'd already tried actionBag.sent and actionBagGrouped::actionBag.sent. No joy. Is this a bug? Seems like others would have stumbled across it already.
> SB
> ________________________________________
> From: Subir S [[EMAIL PROTECTED]]
> Sent: Saturday, June 02, 2012 4:04 AM
> To: [EMAIL PROTECTED]
> Subject: Re: this nested foreach worked in 0.8.1, no longer works in 0.10
>
> Can you try sent = FILTER actionBag BY actionBag.action=='sent';
>
> Although not sure why it does not work. I also use pig 0.8.1 and similar constructs work for me.
>
> On Fri, Jun 1, 2012 at 10:55 PM, Steve Bernstein
> <[EMAIL PROTECTED]>wrote:
>
>> No, all those relations embedded in the nested foreach are unique to
>> it, not used elsewhere.
>> SB
>>
>> -----Original Message-----
>> From: Dmitriy Ryaboy [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, June 01, 2012 2:03 AM
>> To: [EMAIL PROTECTED]
>> Cc: [EMAIL PROTECTED]
>> Subject: Re: this nested foreach worked in 0.8.1, no longer works in
>> 0.10
>>
>> Do you have relations ned sent, opened, etc, elsewhere in the script?
>>
>> On May 31, 2012, at 12:09 PM, Steve Bernstein
>> <[EMAIL PROTECTED]>
>> wrote:
>>
>>> We upgraded from Pig 0.8.1 to 0.10 and the following nested foreach
>>> no
>> longer works:
>>>
>>> actionBagGrouped = GROUP actionBag BY (deal_id,month); dealCounts =
>>> FOREACH actionBagGrouped {
>>> sent = FILTER actionBag BY action=='sent';
>>> opened = FILTER actionBag BY action=='opened';
>>> clicked = FILTER actionBag BY action=='clicked';
>>> purchased = FILTER actionBag BY action=='purchased';
>>>
>>> GENERATE
>>> FLATTEN(group),
>>> COUNT(sent) AS sentCount,
>>> COUNT(opened) AS openedCount,
>>> COUNT(clicked) AS clickedCount,
>>> COUNT(purchased) AS purchasedCount;
>>> }; I get:
>>> "Invalid scalar projection: action : A column needs to be projected
>>> from
>> a relation for it to be used as a scalar"
>>> Banging my head against a wall-the problem is with the nest op sent
>>> >> Filter...
>>>
>>> Help!
>>> Thanks-
>>> Steve
>>

RE: this nested foreach worked in 0.8.1, no longer works in 0.10

Steve Bernstein 2012-06-07, 17:35

Thanks Dmitriy. The full script is probably more than you want (and more than I think I'd want to send), but let me create a "subset" of it, run it to check that it's otherwise sound and that the error replicates, then send that. Would that be okay?

-----Original Message-----
From: Dmitriy Ryaboy [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, June 06, 2012 11:56 AM
To: [EMAIL PROTECTED]
Subject: Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Any chance you can send the full script?

On Jun 6, 2012, at 11:44 AM, Steve Bernstein <[EMAIL PROTECTED]> wrote:

> Meanwhile, I'm moving forward running this script on 0.8.1 with no issues. Probably going to back off on using 0.10 for now and wait for a more stable release.
>
> -----Original Message-----
> From: Steve Bernstein [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, June 03, 2012 9:25 AM
> To: [EMAIL PROTECTED]
> Subject: RE: this nested foreach worked in 0.8.1, no longer works in
> 0.10
>
> Since this seems to do with project, before I wrote to the list, I'd already tried actionBag.sent and actionBagGrouped::actionBag.sent. No joy. Is this a bug? Seems like others would have stumbled across it already.
> SB
> ________________________________________
> From: Subir S [[EMAIL PROTECTED]]
> Sent: Saturday, June 02, 2012 4:04 AM
> To: [EMAIL PROTECTED]
> Subject: Re: this nested foreach worked in 0.8.1, no longer works in
> 0.10
>
> Can you try sent = FILTER actionBag BY actionBag.action=='sent';
>
> Although not sure why it does not work. I also use pig 0.8.1 and similar constructs work for me.
>
> On Fri, Jun 1, 2012 at 10:55 PM, Steve Bernstein
> <[EMAIL PROTECTED]>wrote:
>
>> No, all those relations embedded in the nested foreach are unique to
>> it, not used elsewhere.
>> SB
>>
>> -----Original Message-----
>> From: Dmitriy Ryaboy [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, June 01, 2012 2:03 AM
>> To: [EMAIL PROTECTED]
>> Cc: [EMAIL PROTECTED]
>> Subject: Re: this nested foreach worked in 0.8.1, no longer works in
>> 0.10
>>
>> Do you have relations ned sent, opened, etc, elsewhere in the script?
>>
>> On May 31, 2012, at 12:09 PM, Steve Bernstein
>> <[EMAIL PROTECTED]>
>> wrote:
>>
>>> We upgraded from Pig 0.8.1 to 0.10 and the following nested foreach
>>> no
>> longer works:
>>>
>>> actionBagGrouped = GROUP actionBag BY (deal_id,month); dealCounts =
>>> FOREACH actionBagGrouped {
>>> sent = FILTER actionBag BY action=='sent';
>>> opened = FILTER actionBag BY action=='opened';
>>> clicked = FILTER actionBag BY action=='clicked';
>>> purchased = FILTER actionBag BY action=='purchased';
>>>
>>> GENERATE
>>> FLATTEN(group),
>>> COUNT(sent) AS sentCount,
>>> COUNT(opened) AS openedCount,
>>> COUNT(clicked) AS clickedCount,
>>> COUNT(purchased) AS purchasedCount; };
>>> I get:
>>> "Invalid scalar projection: action : A column needs to be projected
>>> from
>> a relation for it to be used as a scalar"
>>> Banging my head against a wall-the problem is with the nest op sent
>>> >> Filter...
>>>
>>> Help!
>>> Thanks-
>>> Steve
>>

Re: this nested foreach worked in 0.8.1, no longer works in 0.10

Dmitriy Ryaboy 2012-06-07, 23:06

Yeah, anything that reproduces the error would be great. Thanks!

D

On Thu, Jun 7, 2012 at 10:35 AM, Steve Bernstein
<[EMAIL PROTECTED]> wrote:
> Thanks Dmitriy. The full script is probably more than you want (and more than I think I'd want to send), but let me create a "subset" of it, run it to check that it's otherwise sound and that the error replicates, then send that. Would that be okay?
>
> -----Original Message-----
> From: Dmitriy Ryaboy [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, June 06, 2012 11:56 AM
> To: [EMAIL PROTECTED]
> Subject: Re: this nested foreach worked in 0.8.1, no longer works in 0.10
>
> Any chance you can send the full script?
>
> On Jun 6, 2012, at 11:44 AM, Steve Bernstein <[EMAIL PROTECTED]> wrote:
>
>> Meanwhile, I'm moving forward running this script on 0.8.1 with no issues. Probably going to back off on using 0.10 for now and wait for a more stable release.
>>
>> -----Original Message-----
>> From: Steve Bernstein [mailto:[EMAIL PROTECTED]]
>> Sent: Sunday, June 03, 2012 9:25 AM
>> To: [EMAIL PROTECTED]
>> Subject: RE: this nested foreach worked in 0.8.1, no longer works in
>> 0.10
>>
>> Since this seems to do with project, before I wrote to the list, I'd already tried actionBag.sent and actionBagGrouped::actionBag.sent. No joy. Is this a bug? Seems like others would have stumbled across it already.
>> SB
>> ________________________________________
>> From: Subir S [[EMAIL PROTECTED]]
>> Sent: Saturday, June 02, 2012 4:04 AM
>> To: [EMAIL PROTECTED]
>> Subject: Re: this nested foreach worked in 0.8.1, no longer works in
>> 0.10
>>
>> Can you try sent = FILTER actionBag BY actionBag.action=='sent';
>>
>> Although not sure why it does not work. I also use pig 0.8.1 and similar constructs work for me.
>>
>> On Fri, Jun 1, 2012 at 10:55 PM, Steve Bernstein
>> <[EMAIL PROTECTED]>wrote:
>>
>>> No, all those relations embedded in the nested foreach are unique to
>>> it, not used elsewhere.
>>> SB
>>>
>>> -----Original Message-----
>>> From: Dmitriy Ryaboy [mailto:[EMAIL PROTECTED]]
>>> Sent: Friday, June 01, 2012 2:03 AM
>>> To: [EMAIL PROTECTED]
>>> Cc: [EMAIL PROTECTED]
>>> Subject: Re: this nested foreach worked in 0.8.1, no longer works in
>>> 0.10
>>>
>>> Do you have relations ned sent, opened, etc, elsewhere in the script?
>>>
>>> On May 31, 2012, at 12:09 PM, Steve Bernstein
>>> <[EMAIL PROTECTED]>
>>> wrote:
>>>
>>>> We upgraded from Pig 0.8.1 to 0.10 and the following nested foreach
>>>> no
>>> longer works:
>>>>
>>>> actionBagGrouped = GROUP actionBag BY (deal_id,month); dealCounts >>>> FOREACH actionBagGrouped {
>>>> sent = FILTER actionBag BY action=='sent';
>>>> opened = FILTER actionBag BY action=='opened';
>>>> clicked = FILTER actionBag BY action=='clicked';
>>>> purchased = FILTER actionBag BY action=='purchased';
>>>>
>>>> GENERATE
>>>> FLATTEN(group),
>>>> COUNT(sent) AS sentCount,
>>>> COUNT(opened) AS openedCount,
>>>> COUNT(clicked) AS clickedCount,
>>>> COUNT(purchased) AS purchasedCount; };
>>>> I get:
>>>> "Invalid scalar projection: action : A column needs to be projected
>>>> from
>>> a relation for it to be used as a scalar"
>>>> Banging my head against a wall-the problem is with the nest op sent
>>>> >>> Filter...
>>>>
>>>> Help!
>>>> Thanks-
>>>> Steve
>>>