Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Inserting date


Copy link to this message
-
Re: Inserting date
Mohit Anchlia 2012-03-23, 01:04
Thanks

On Thu, Mar 22, 2012 at 5:09 PM, Thejas Nair <[EMAIL PROTECTED]> wrote:

> On 3/22/12 4:18 PM, Mohit Anchlia wrote:
>
>> Thanks! Is there any time function that I can use in pig to get the
>> current
>> time?
>>
>
> There might be a date udf you can use. But as you probably want same date
> for all data (if your query happens to runs across date boundary, or a node
> has a problem) the better option is to use parameter substitution in pig.
>
> Change the line to -
> B = foreach A generate *, '$DATE' as date;
>
> Then specify the date param your pig command, using the *nix date cmd  -
>
> pig -p DATE=`date +%Y%m%d` query.pig
>
> -Thejas
>
>
>
>
>
>> On Thu, Mar 22, 2012 at 3:58 PM, Jonathan Coveney<[EMAIL PROTECTED]>**
>> wrote:
>>
>> There is not going to be an additional cost to adding the date as Thejas
>>> posted. If that's what you want, do it.
>>>
>>> Optionally, you could extend your StoreFunc to prepend the date to every
>>> record it writes. I assure you, doing what Thejas said is simpler (or
>>> just
>>> having the database you write to populate the date for you!)
>>>
>>> 2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]**>
>>>
>>> On Thu, Mar 22, 2012 at 2:34 PM, Thejas Nair<[EMAIL PROTECTED]>
>>>> wrote:
>>>>
>>>> Is this what you are looking for  ? -
>>>>>
>>>>>
>>>>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>>>>
>>>>> B = foreach A generate *, '20120322' as date;
>>>>>
>>>>> STORE B into ...
>>>>>
>>>>> Thanks,
>>>>> Thejas
>>>>>
>>>>
>>>>
>>>> this I am assuming will run another map task? I was trying to avoid
>>>> that.
>>>>
>>>>
>>>>>
>>>>>
>>>>> On 3/22/12 1:13 PM, Mohit Anchlia wrote:
>>>>>
>>>>> Yes that's exactly what I am asking. Reading from flat file and then
>>>>>> inserting it into the database. And I want to insert date before
>>>>>>
>>>>> storing.
>>>>
>>>>>
>>>>>> for eg I want to add date before A gets stored:
>>>>>>
>>>>>>
>>>>>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>>>>>
>>>>>> STORE A into ...
>>>>>>
>>>>>> On Thu, Mar 22, 2012 at 12:54 PM, Jonathan Coveney<[EMAIL PROTECTED]
>>>>>>
>>>>> **
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> Do you mean you're reading a relation from Hadoop, and want to append
>>>>>>
>>>>> the
>>>>
>>>>>  date to the row before inserting it? I'm not quite sure what you're
>>>>>>> asking
>>>>>>> for.
>>>>>>>
>>>>>>>  >  >>>  2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]****>
>>>
>>>>
>>>>>>> Sorry I mean to ask if there is any way to insert date into the ALIAS
>>>>>>>
>>>>>> so
>>>>
>>>>>  that I can use it before storing it into DB.
>>>>>>>>
>>>>>>>> On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia<
>>>>>>>>
>>>>>>> [EMAIL PROTECTED]
>>>>
>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I am reading bunch of columns from a flat file and inserting it into
>>>>>>>>
>>>>>>>>>
>>>>>>>>> the
>>>>>>>>
>>>>>>>
>>>>>>> database. Is there a way to also insert date?
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>