Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Inserting date


Copy link to this message
-
Re: Inserting date
On 3/22/12 4:18 PM, Mohit Anchlia wrote:
> Thanks! Is there any time function that I can use in pig to get the current
> time?

There might be a date udf you can use. But as you probably want same
date for all data (if your query happens to runs across date boundary,
or a node has a problem) the better option is to use parameter
substitution in pig.

Change the line to -
B = foreach A generate *, '$DATE' as date;

Then specify the date param your pig command, using the *nix date cmd  -

pig -p DATE=`date +%Y%m%d` query.pig

-Thejas

>
> On Thu, Mar 22, 2012 at 3:58 PM, Jonathan Coveney<[EMAIL PROTECTED]>wrote:
>
>> There is not going to be an additional cost to adding the date as Thejas
>> posted. If that's what you want, do it.
>>
>> Optionally, you could extend your StoreFunc to prepend the date to every
>> record it writes. I assure you, doing what Thejas said is simpler (or just
>> having the database you write to populate the date for you!)
>>
>> 2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]>
>>
>>> On Thu, Mar 22, 2012 at 2:34 PM, Thejas Nair<[EMAIL PROTECTED]>
>>> wrote:
>>>
>>>> Is this what you are looking for  ? -
>>>>
>>>>
>>>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>>>
>>>> B = foreach A generate *, '20120322' as date;
>>>>
>>>> STORE B into ...
>>>>
>>>> Thanks,
>>>> Thejas
>>>
>>>
>>> this I am assuming will run another map task? I was trying to avoid that.
>>>
>>>>
>>>>
>>>>
>>>> On 3/22/12 1:13 PM, Mohit Anchlia wrote:
>>>>
>>>>> Yes that's exactly what I am asking. Reading from flat file and then
>>>>> inserting it into the database. And I want to insert date before
>>> storing.
>>>>>
>>>>> for eg I want to add date before A gets stored:
>>>>>
>>>>>
>>>>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>>>>
>>>>> STORE A into ...
>>>>>
>>>>> On Thu, Mar 22, 2012 at 12:54 PM, Jonathan Coveney<[EMAIL PROTECTED]
>>>> **
>>>>> wrote:
>>>>>
>>>>> Do you mean you're reading a relation from Hadoop, and want to append
>>> the
>>>>>> date to the row before inserting it? I'm not quite sure what you're
>>>>>> asking
>>>>>> for.
>>>>>>
>>   >  >>>  2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]**>
>>>>>>
>>>>>> Sorry I mean to ask if there is any way to insert date into the ALIAS
>>> so
>>>>>>> that I can use it before storing it into DB.
>>>>>>>
>>>>>>> On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia<
>>> [EMAIL PROTECTED]
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>> I am reading bunch of columns from a flat file and inserting it into
>>>>>>>>
>>>>>>> the
>>>>>>
>>>>>>> database. Is there a way to also insert date?
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>