Pig, mail # user - Multiple CurrentTime calls return the same timestamp

Multiple CurrentTime calls return the same timestamp

Danfeng Li 2013-02-27, 00:09

I called CurrentTime() twice in my pig code, but the final results are end up as the same timestamp.

The code is following:

A = load 'test.txt' as (a:chararray);
dump A;
B = foreach A generate a, CurrentTime();
dump B;
C = foreach B generate a as a,
$1 as time1,
CurrentTime() as time2;
dump C;

The results are

A:
(adg_daadf)
(test_Test)

B:
(adg_daadf,2013-02-26T16:04:55.049-08:00)
(test_Test,2013-02-26T16:04:55.049-08:00)

C:
(adg_daadf,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:00)
(test_Test,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:00)

Note how the time1 and time2 in "C" are the same, and time1 is different from the time in B.

Are this a bug or a feature?

If this is a feature, how do I preserve the timestamp in B.

Thanks.
Dan

Re: Multiple CurrentTime calls return the same timestamp

Cheolsoo Park 2013-02-27, 00:51

Hi Dan,

Are you using 0.11 or trunk?

If you're using trunk, please take a look at PIG-3014.
https://issues.apache.org/jira/browse/PIG-3014

Thanks,
Cheolsoo

On Tue, Feb 26, 2013 at 4:09 PM, Danfeng Li <[EMAIL PROTECTED]> wrote:

> I called CurrentTime() twice in my pig code, but the final results are end
> up as the same timestamp.
>
> The code is following:
>
> A = load 'test.txt' as (a:chararray);
> dump A;
> B = foreach A generate a, CurrentTime();
> dump B;
> C = foreach B generate a as a,
> $1 as time1,
> CurrentTime() as time2;
> dump C;
>
> The results are
>
> A:
> (adg_daadf)
> (test_Test)
>
> B:
> (adg_daadf,2013-02-26T16:04:55.049-08:00)
> (test_Test,2013-02-26T16:04:55.049-08:00)
>
> C:
> (adg_daadf,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:00)
> (test_Test,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:00)
>
> Note how the time1 and time2 in "C" are the same, and time1 is different
> from the time in B.
>
> Are this a bug or a feature?
>
> If this is a feature, how do I preserve the timestamp in B.
>
> Thanks.
> Dan
>
>

Re: Multiple CurrentTime calls return the same timestamp

Jonathan Coveney 2013-02-27, 09:30

This is by design, as the notion of a CurrentTime() in a Pig job is a big
poorly specified, so we went with something "unremarkable." What do you
think it should be?
2013/2/27 Cheolsoo Park <[EMAIL PROTECTED]>

> Hi Dan,
>
> Are you using 0.11 or trunk?
>
> If you're using trunk, please take a look at PIG-3014.
> https://issues.apache.org/jira/browse/PIG-3014
>
> Thanks,
> Cheolsoo
>
>
>
> On Tue, Feb 26, 2013 at 4:09 PM, Danfeng Li <[EMAIL PROTECTED]>
> wrote:
>
> > I called CurrentTime() twice in my pig code, but the final results are
> end
> > up as the same timestamp.
> >
> > The code is following:
> >
> > A = load 'test.txt' as (a:chararray);
> > dump A;
> > B = foreach A generate a, CurrentTime();
> > dump B;
> > C = foreach B generate a as a,
> > $1 as time1,
> > CurrentTime() as time2;
> > dump C;
> >
> > The results are
> >
> > A:
> > (adg_daadf)
> > (test_Test)
> >
> > B:
> > (adg_daadf,2013-02-26T16:04:55.049-08:00)
> > (test_Test,2013-02-26T16:04:55.049-08:00)
> >
> > C:
> > (adg_daadf,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:00)
> > (test_Test,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:00)
> >
> > Note how the time1 and time2 in "C" are the same, and time1 is different
> > from the time in B.
> >
> > Are this a bug or a feature?
> >
> > If this is a feature, how do I preserve the timestamp in B.
> >
> > Thanks.
> > Dan
> >
> >
>

RE: Multiple CurrentTime calls return the same timestamp

Danfeng Li 2013-02-27, 19:02

So if I understand it correctly, for the same run, all the CurrentTime() will return the same timestamp. I wonder if there any udf can provide runtime timestamp.

Thanks.
Dan

-----Original Message-----
From: Jonathan Coveney [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, February 27, 2013 1:30 AM
To: [EMAIL PROTECTED]
Subject: Re: Multiple CurrentTime calls return the same timestamp

This is by design, as the notion of a CurrentTime() in a Pig job is a big poorly specified, so we went with something "unremarkable." What do you think it should be?
2013/2/27 Cheolsoo Park <[EMAIL PROTECTED]>

> Hi Dan,
>
> Are you using 0.11 or trunk?
>
> If you're using trunk, please take a look at PIG-3014.
> https://issues.apache.org/jira/browse/PIG-3014
>
> Thanks,
> Cheolsoo
>
>
>
> On Tue, Feb 26, 2013 at 4:09 PM, Danfeng Li <[EMAIL PROTECTED]>
> wrote:
>
> > I called CurrentTime() twice in my pig code, but the final results
> > are
> end
> > up as the same timestamp.
> >
> > The code is following:
> >
> > A = load 'test.txt' as (a:chararray); dump A; B = foreach A generate
> > a, CurrentTime(); dump B; C = foreach B generate a as a,
> > $1 as time1,
> > CurrentTime() as time2;
> > dump C;
> >
> > The results are
> >
> > A:
> > (adg_daadf)
> > (test_Test)
> >
> > B:
> > (adg_daadf,2013-02-26T16:04:55.049-08:00)
> > (test_Test,2013-02-26T16:04:55.049-08:00)
> >
> > C:
> > (adg_daadf,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:
> > 00)
> > (test_Test,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:
> > 00)
> >
> > Note how the time1 and time2 in "C" are the same, and time1 is
> > different from the time in B.
> >
> > Are this a bug or a feature?
> >
> > If this is a feature, how do I preserve the timestamp in B.
> >
> > Thanks.
> > Dan
> >
> >
>

Re: Multiple CurrentTime calls return the same timestamp

Jonathan Coveney 2013-02-27, 22:49

What is a meaningful definition of runtime timestamp? Is it when an
individual mapper runs? What happens if mappers are scheduled at very
different times? What happens if a mapper fails and is restarted?

What is the end result that you would like?
2013/2/27 Danfeng Li <[EMAIL PROTECTED]>

> So if I understand it correctly, for the same run, all the CurrentTime()
> will return the same timestamp. I wonder if there any udf can provide
> runtime timestamp.
>
> Thanks.
> Dan
>
> -----Original Message-----
> From: Jonathan Coveney [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, February 27, 2013 1:30 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Multiple CurrentTime calls return the same timestamp
>
> This is by design, as the notion of a CurrentTime() in a Pig job is a big
> poorly specified, so we went with something "unremarkable." What do you
> think it should be?
>
>
> 2013/2/27 Cheolsoo Park <[EMAIL PROTECTED]>
>
> > Hi Dan,
> >
> > Are you using 0.11 or trunk?
> >
> > If you're using trunk, please take a look at PIG-3014.
> > https://issues.apache.org/jira/browse/PIG-3014
> >
> > Thanks,
> > Cheolsoo
> >
> >
> >
> > On Tue, Feb 26, 2013 at 4:09 PM, Danfeng Li <[EMAIL PROTECTED]>
> > wrote:
> >
> > > I called CurrentTime() twice in my pig code, but the final results
> > > are
> > end
> > > up as the same timestamp.
> > >
> > > The code is following:
> > >
> > > A = load 'test.txt' as (a:chararray); dump A; B = foreach A generate
> > > a, CurrentTime(); dump B; C = foreach B generate a as a,
> > > $1 as time1,
> > > CurrentTime() as time2;
> > > dump C;
> > >
> > > The results are
> > >
> > > A:
> > > (adg_daadf)
> > > (test_Test)
> > >
> > > B:
> > > (adg_daadf,2013-02-26T16:04:55.049-08:00)
> > > (test_Test,2013-02-26T16:04:55.049-08:00)
> > >
> > > C:
> > > (adg_daadf,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:
> > > 00)
> > > (test_Test,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:
> > > 00)
> > >
> > > Note how the time1 and time2 in "C" are the same, and time1 is
> > > different from the time in B.
> > >
> > > Are this a bug or a feature?
> > >
> > > If this is a feature, how do I preserve the timestamp in B.
> > >
> > > Thanks.
> > > Dan
> > >
> > >
> >
>