Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Multiple CurrentTime calls return the same timestamp


Copy link to this message
-
Re: Multiple CurrentTime calls return the same timestamp
Jonathan Coveney 2013-02-27, 22:49
What is a meaningful definition of runtime timestamp? Is it when an
individual mapper runs? What happens if mappers are scheduled at very
different times? What happens if a mapper fails and is restarted?

What is the end result that you would like?
2013/2/27 Danfeng Li <[EMAIL PROTECTED]>

> So if I understand it correctly, for the same run, all the CurrentTime()
> will return the same timestamp. I wonder if there any udf can provide
> runtime timestamp.
>
> Thanks.
> Dan
>
> -----Original Message-----
> From: Jonathan Coveney [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, February 27, 2013 1:30 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Multiple CurrentTime calls return the same timestamp
>
> This is by design, as the notion of a CurrentTime() in a Pig job is a big
> poorly specified, so we went with something "unremarkable." What do you
> think it should be?
>
>
> 2013/2/27 Cheolsoo Park <[EMAIL PROTECTED]>
>
> > Hi Dan,
> >
> > Are you using 0.11 or trunk?
> >
> > If you're using trunk, please take a look at PIG-3014.
> > https://issues.apache.org/jira/browse/PIG-3014
> >
> > Thanks,
> > Cheolsoo
> >
> >
> >
> > On Tue, Feb 26, 2013 at 4:09 PM, Danfeng Li <[EMAIL PROTECTED]>
> > wrote:
> >
> > > I called CurrentTime() twice in my pig code, but the final results
> > > are
> > end
> > > up as the same timestamp.
> > >
> > > The code is following:
> > >
> > > A = load 'test.txt' as (a:chararray); dump A; B = foreach A generate
> > > a, CurrentTime(); dump B; C = foreach B generate a as a,
> > >     $1 as time1,
> > >     CurrentTime() as time2;
> > > dump C;
> > >
> > > The results are
> > >
> > > A:
> > > (adg_daadf)
> > > (test_Test)
> > >
> > > B:
> > > (adg_daadf,2013-02-26T16:04:55.049-08:00)
> > > (test_Test,2013-02-26T16:04:55.049-08:00)
> > >
> > > C:
> > > (adg_daadf,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:
> > > 00)
> > > (test_Test,2013-02-26T16:05:08.979-08:00,2013-02-26T16:05:08.979-08:
> > > 00)
> > >
> > > Note how the time1 and time2 in "C" are the same, and time1 is
> > > different from the time in B.
> > >
> > > Are this a bug or a feature?
> > >
> > > If this is a feature, how do I preserve the timestamp in B.
> > >
> > > Thanks.
> > > Dan
> > >
> > >
> >
>