Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Inserting date


Copy link to this message
-
Re: Inserting date
Thanks

On Thu, Mar 22, 2012 at 5:09 PM, Thejas Nair <[EMAIL PROTECTED]> wrote:

> On 3/22/12 4:18 PM, Mohit Anchlia wrote:
>
>> Thanks! Is there any time function that I can use in pig to get the
>> current
>> time?
>>
>
> There might be a date udf you can use. But as you probably want same date
> for all data (if your query happens to runs across date boundary, or a node
> has a problem) the better option is to use parameter substitution in pig.
>
> Change the line to -
> B = foreach A generate *, '$DATE' as date;
>
> Then specify the date param your pig command, using the *nix date cmd  -
>
> pig -p DATE=`date +%Y%m%d` query.pig
>
> -Thejas
>
>
>
>
>
>> On Thu, Mar 22, 2012 at 3:58 PM, Jonathan Coveney<[EMAIL PROTECTED]>**
>> wrote:
>>
>> There is not going to be an additional cost to adding the date as Thejas
>>> posted. If that's what you want, do it.
>>>
>>> Optionally, you could extend your StoreFunc to prepend the date to every
>>> record it writes. I assure you, doing what Thejas said is simpler (or
>>> just
>>> having the database you write to populate the date for you!)
>>>
>>> 2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]**>
>>>
>>> On Thu, Mar 22, 2012 at 2:34 PM, Thejas Nair<[EMAIL PROTECTED]>
>>>> wrote:
>>>>
>>>> Is this what you are looking for  ? -
>>>>>
>>>>>
>>>>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>>>>
>>>>> B = foreach A generate *, '20120322' as date;
>>>>>
>>>>> STORE B into ...
>>>>>
>>>>> Thanks,
>>>>> Thejas
>>>>>
>>>>
>>>>
>>>> this I am assuming will run another map task? I was trying to avoid
>>>> that.
>>>>
>>>>
>>>>>
>>>>>
>>>>> On 3/22/12 1:13 PM, Mohit Anchlia wrote:
>>>>>
>>>>> Yes that's exactly what I am asking. Reading from flat file and then
>>>>>> inserting it into the database. And I want to insert date before
>>>>>>
>>>>> storing.
>>>>
>>>>>
>>>>>> for eg I want to add date before A gets stored:
>>>>>>
>>>>>>
>>>>>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>>>>>
>>>>>> STORE A into ...
>>>>>>
>>>>>> On Thu, Mar 22, 2012 at 12:54 PM, Jonathan Coveney<[EMAIL PROTECTED]
>>>>>>
>>>>> **
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> Do you mean you're reading a relation from Hadoop, and want to append
>>>>>>
>>>>> the
>>>>
>>>>>  date to the row before inserting it? I'm not quite sure what you're
>>>>>>> asking
>>>>>>> for.
>>>>>>>
>>>>>>>  >  >>>  2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]****>
>>>
>>>>
>>>>>>> Sorry I mean to ask if there is any way to insert date into the ALIAS
>>>>>>>
>>>>>> so
>>>>
>>>>>  that I can use it before storing it into DB.
>>>>>>>>
>>>>>>>> On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia<
>>>>>>>>
>>>>>>> [EMAIL PROTECTED]
>>>>
>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I am reading bunch of columns from a flat file and inserting it into
>>>>>>>>
>>>>>>>>>
>>>>>>>>> the
>>>>>>>>
>>>>>>>
>>>>>>> database. Is there a way to also insert date?
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB