I am reading bunch of columns from a flat file and inserting it into the
database. Is there a way to also insert date?

Sorry I mean to ask if there is any way to insert date into the ALIAS so
that I can use it before storing it into DB.

On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:

> I am reading bunch of columns from a flat file and inserting it into the
> database. Is there a way to also insert date?
>

Do you mean you're reading a relation from Hadoop, and want to append the
date to the row before inserting it? I'm not quite sure what you're asking
for.

2012/3/22 Mohit Anchlia <[EMAIL PROTECTED]>

> Sorry I mean to ask if there is any way to insert date into the ALIAS so
> that I can use it before storing it into DB.
>
> On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia <[EMAIL PROTECTED]
> >wrote:
>
> > I am reading bunch of columns from a flat file and inserting it into the
> > database. Is there a way to also insert date?
> >
>

Yes that's exactly what I am asking. Reading from flat file and then
inserting it into the database. And I want to insert date before storing.

for eg I want to add date before A gets stored:
A = LOAD '$in' USING PigStorage('\t') AS (...

STORE A into ...

On Thu, Mar 22, 2012 at 12:54 PM, Jonathan Coveney <[EMAIL PROTECTED]>wrote:

> Do you mean you're reading a relation from Hadoop, and want to append the
> date to the row before inserting it? I'm not quite sure what you're asking
> for.
>
> 2012/3/22 Mohit Anchlia <[EMAIL PROTECTED]>
>
> > Sorry I mean to ask if there is any way to insert date into the ALIAS so
> > that I can use it before storing it into DB.
> >
> > On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia <[EMAIL PROTECTED]
> > >wrote:
> >
> > > I am reading bunch of columns from a flat file and inserting it into
> the
> > > database. Is there a way to also insert date?
> > >
> >
>

Is this what you are looking for  ? -

A = LOAD '$in' USING PigStorage('\t') AS (...

B = foreach A generate *, '20120322' as date;

STORE B into ...

Thanks,
Thejas
On 3/22/12 1:13 PM, Mohit Anchlia wrote:
> Yes that's exactly what I am asking. Reading from flat file and then
> inserting it into the database. And I want to insert date before storing.
>
> for eg I want to add date before A gets stored:
>
>
> A = LOAD '$in' USING PigStorage('\t') AS (...
>
> STORE A into ...
>
> On Thu, Mar 22, 2012 at 12:54 PM, Jonathan Coveney<[EMAIL PROTECTED]>wrote:
>
>> Do you mean you're reading a relation from Hadoop, and want to append the
>> date to the row before inserting it? I'm not quite sure what you're asking
>> for.
>>
>> 2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]>
>>
>>> Sorry I mean to ask if there is any way to insert date into the ALIAS so
>>> that I can use it before storing it into DB.
>>>
>>> On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia<[EMAIL PROTECTED]
>>>> wrote:
>>>
>>>> I am reading bunch of columns from a flat file and inserting it into
>> the
>>>> database. Is there a way to also insert date?
>>>>
>>>
>>
>

Mohit,

Is date a field in your dataset, or current date or something else? Few
options
   1. You could let Database implicitly create a date field if you need the
   INSERT date
   2. As Thejas suggested, simply insert it as '20120322' as date. I don't
   think DB has any dependency on understanding Pig types. It would be similar
   to loading a CSV file into a database.

Thanks,

Prashant

On Thu, Mar 22, 2012 at 2:34 PM, Thejas Nair <[EMAIL PROTECTED]> wrote:

> Is this what you are looking for  ? -
>
>
> A = LOAD '$in' USING PigStorage('\t') AS (...
>
> B = foreach A generate *, '20120322' as date;
>
> STORE B into ...
>
> Thanks,
> Thejas
>
>
>
> On 3/22/12 1:13 PM, Mohit Anchlia wrote:
>
>> Yes that's exactly what I am asking. Reading from flat file and then
>> inserting it into the database. And I want to insert date before storing.
>>
>> for eg I want to add date before A gets stored:
>>
>>
>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>
>> STORE A into ...
>>
>> On Thu, Mar 22, 2012 at 12:54 PM, Jonathan Coveney<[EMAIL PROTECTED]>**
>> wrote:
>>
>>  Do you mean you're reading a relation from Hadoop, and want to append the
>>> date to the row before inserting it? I'm not quite sure what you're
>>> asking
>>> for.
>>>
>>> 2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]**>
>>>
>>>  Sorry I mean to ask if there is any way to insert date into the ALIAS so
>>>> that I can use it before storing it into DB.
>>>>
>>>> On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia<[EMAIL PROTECTED]
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>>  I am reading bunch of columns from a flat file and inserting it into
>>>>>
>>>> the
>>>
>>>> database. Is there a way to also insert date?
>>>>>
>>>>>
>>>>
>>>
>>
>

On Thu, Mar 22, 2012 at 2:34 PM, Thejas Nair <[EMAIL PROTECTED]> wrote:

> Is this what you are looking for  ? -
>
>
> A = LOAD '$in' USING PigStorage('\t') AS (...
>
> B = foreach A generate *, '20120322' as date;
>
> STORE B into ...
>
> Thanks,
> Thejas
this I am assuming will run another map task? I was trying to avoid that.

>
>
>
> On 3/22/12 1:13 PM, Mohit Anchlia wrote:
>
>> Yes that's exactly what I am asking. Reading from flat file and then
>> inserting it into the database. And I want to insert date before storing.
>>
>> for eg I want to add date before A gets stored:
>>
>>
>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>
>> STORE A into ...
>>
>> On Thu, Mar 22, 2012 at 12:54 PM, Jonathan Coveney<[EMAIL PROTECTED]>**
>> wrote:
>>
>> Do you mean you're reading a relation from Hadoop, and want to append the
>>> date to the row before inserting it? I'm not quite sure what you're
>>> asking
>>> for.
>>>
>>> 2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]**>
>>>
>>> Sorry I mean to ask if there is any way to insert date into the ALIAS so
>>>> that I can use it before storing it into DB.
>>>>
>>>> On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia<[EMAIL PROTECTED]
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> I am reading bunch of columns from a flat file and inserting it into
>>>>>
>>>> the
>>>
>>>> database. Is there a way to also insert date?
>>>>>
>>>>>
>>>>
>>>
>>
>

There is not going to be an additional cost to adding the date as Thejas
posted. If that's what you want, do it.

Optionally, you could extend your StoreFunc to prepend the date to every
record it writes. I assure you, doing what Thejas said is simpler (or just
having the database you write to populate the date for you!)

2012/3/22 Mohit Anchlia <[EMAIL PROTECTED]>

> On Thu, Mar 22, 2012 at 2:34 PM, Thejas Nair <[EMAIL PROTECTED]>
> wrote:
>
> > Is this what you are looking for  ? -
> >
> >
> > A = LOAD '$in' USING PigStorage('\t') AS (...
> >
> > B = foreach A generate *, '20120322' as date;
> >
> > STORE B into ...
> >
> > Thanks,
> > Thejas
>
>
> this I am assuming will run another map task? I was trying to avoid that.
>
> >
> >
> >
> > On 3/22/12 1:13 PM, Mohit Anchlia wrote:
> >
> >> Yes that's exactly what I am asking. Reading from flat file and then
> >> inserting it into the database. And I want to insert date before
> storing.
> >>
> >> for eg I want to add date before A gets stored:
> >>
> >>
> >> A = LOAD '$in' USING PigStorage('\t') AS (...
> >>
> >> STORE A into ...
> >>
> >> On Thu, Mar 22, 2012 at 12:54 PM, Jonathan Coveney<[EMAIL PROTECTED]
> >**
> >> wrote:
> >>
> >> Do you mean you're reading a relation from Hadoop, and want to append
> the
> >>> date to the row before inserting it? I'm not quite sure what you're
> >>> asking
> >>> for.
> >>>
> >>> 2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]**>
> >>>
> >>> Sorry I mean to ask if there is any way to insert date into the ALIAS
> so
> >>>> that I can use it before storing it into DB.
> >>>>
> >>>> On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia<
> [EMAIL PROTECTED]
> >>>>
> >>>>> wrote:
> >>>>>
> >>>>
> >>>> I am reading bunch of columns from a flat file and inserting it into
> >>>>>
> >>>> the
> >>>
> >>>> database. Is there a way to also insert date?
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>

Thanks! Is there any time function that I can use in pig to get the current
time?

On Thu, Mar 22, 2012 at 3:58 PM, Jonathan Coveney <[EMAIL PROTECTED]>wrote:

> There is not going to be an additional cost to adding the date as Thejas
> posted. If that's what you want, do it.
>
> Optionally, you could extend your StoreFunc to prepend the date to every
> record it writes. I assure you, doing what Thejas said is simpler (or just
> having the database you write to populate the date for you!)
>
> 2012/3/22 Mohit Anchlia <[EMAIL PROTECTED]>
>
> > On Thu, Mar 22, 2012 at 2:34 PM, Thejas Nair <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Is this what you are looking for  ? -
> > >
> > >
> > > A = LOAD '$in' USING PigStorage('\t') AS (...
> > >
> > > B = foreach A generate *, '20120322' as date;
> > >
> > > STORE B into ...
> > >
> > > Thanks,
> > > Thejas
> >
> >
> > this I am assuming will run another map task? I was trying to avoid that.
> >
> > >
> > >
> > >
> > > On 3/22/12 1:13 PM, Mohit Anchlia wrote:
> > >
> > >> Yes that's exactly what I am asking. Reading from flat file and then
> > >> inserting it into the database. And I want to insert date before
> > storing.
> > >>
> > >> for eg I want to add date before A gets stored:
> > >>
> > >>
> > >> A = LOAD '$in' USING PigStorage('\t') AS (...
> > >>
> > >> STORE A into ...
> > >>
> > >> On Thu, Mar 22, 2012 at 12:54 PM, Jonathan Coveney<[EMAIL PROTECTED]
> > >**
> > >> wrote:
> > >>
> > >> Do you mean you're reading a relation from Hadoop, and want to append
> > the
> > >>> date to the row before inserting it? I'm not quite sure what you're
> > >>> asking
> > >>> for.
> > >>>
>  > >>> 2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]**>
> > >>>
> > >>> Sorry I mean to ask if there is any way to insert date into the ALIAS
> > so
> > >>>> that I can use it before storing it into DB.
> > >>>>
> > >>>> On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia<
> > [EMAIL PROTECTED]
> > >>>>
> > >>>>> wrote:
> > >>>>>
> > >>>>
> > >>>> I am reading bunch of columns from a flat file and inserting it into
> > >>>>>
> > >>>> the
> > >>>
> > >>>> database. Is there a way to also insert date?
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >
> >
>

On 3/22/12 4:18 PM, Mohit Anchlia wrote:
> Thanks! Is there any time function that I can use in pig to get the current
> time?

There might be a date udf you can use. But as you probably want same
date for all data (if your query happens to runs across date boundary,
or a node has a problem) the better option is to use parameter
substitution in pig.

Change the line to -
B = foreach A generate *, '$DATE' as date;

Then specify the date param your pig command, using the *nix date cmd  -

pig -p DATE=`date +%Y%m%d` query.pig

-Thejas

>
> On Thu, Mar 22, 2012 at 3:58 PM, Jonathan Coveney<[EMAIL PROTECTED]>wrote:
>
>> There is not going to be an additional cost to adding the date as Thejas
>> posted. If that's what you want, do it.
>>
>> Optionally, you could extend your StoreFunc to prepend the date to every
>> record it writes. I assure you, doing what Thejas said is simpler (or just
>> having the database you write to populate the date for you!)
>>
>> 2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]>
>>
>>> On Thu, Mar 22, 2012 at 2:34 PM, Thejas Nair<[EMAIL PROTECTED]>
>>> wrote:
>>>
>>>> Is this what you are looking for  ? -
>>>>
>>>>
>>>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>>>
>>>> B = foreach A generate *, '20120322' as date;
>>>>
>>>> STORE B into ...
>>>>
>>>> Thanks,
>>>> Thejas
>>>
>>>
>>> this I am assuming will run another map task? I was trying to avoid that.
>>>
>>>>
>>>>
>>>>
>>>> On 3/22/12 1:13 PM, Mohit Anchlia wrote:
>>>>
>>>>> Yes that's exactly what I am asking. Reading from flat file and then
>>>>> inserting it into the database. And I want to insert date before
>>> storing.
>>>>>
>>>>> for eg I want to add date before A gets stored:
>>>>>
>>>>>
>>>>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>>>>
>>>>> STORE A into ...
>>>>>
>>>>> On Thu, Mar 22, 2012 at 12:54 PM, Jonathan Coveney<[EMAIL PROTECTED]
>>>> **
>>>>> wrote:
>>>>>
>>>>> Do you mean you're reading a relation from Hadoop, and want to append
>>> the
>>>>>> date to the row before inserting it? I'm not quite sure what you're
>>>>>> asking
>>>>>> for.
>>>>>>
>>   >  >>>  2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]**>
>>>>>>
>>>>>> Sorry I mean to ask if there is any way to insert date into the ALIAS
>>> so
>>>>>>> that I can use it before storing it into DB.
>>>>>>>
>>>>>>> On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia<
>>> [EMAIL PROTECTED]
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>> I am reading bunch of columns from a flat file and inserting it into
>>>>>>>>
>>>>>>> the
>>>>>>
>>>>>>> database. Is there a way to also insert date?
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Thanks

On Thu, Mar 22, 2012 at 5:09 PM, Thejas Nair <[EMAIL PROTECTED]> wrote:

> On 3/22/12 4:18 PM, Mohit Anchlia wrote:
>
>> Thanks! Is there any time function that I can use in pig to get the
>> current
>> time?
>>
>
> There might be a date udf you can use. But as you probably want same date
> for all data (if your query happens to runs across date boundary, or a node
> has a problem) the better option is to use parameter substitution in pig.
>
> Change the line to -
> B = foreach A generate *, '$DATE' as date;
>
> Then specify the date param your pig command, using the *nix date cmd  -
>
> pig -p DATE=`date +%Y%m%d` query.pig
>
> -Thejas
>
>
>
>
>
>> On Thu, Mar 22, 2012 at 3:58 PM, Jonathan Coveney<[EMAIL PROTECTED]>**
>> wrote:
>>
>> There is not going to be an additional cost to adding the date as Thejas
>>> posted. If that's what you want, do it.
>>>
>>> Optionally, you could extend your StoreFunc to prepend the date to every
>>> record it writes. I assure you, doing what Thejas said is simpler (or
>>> just
>>> having the database you write to populate the date for you!)
>>>
>>> 2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]**>
>>>
>>> On Thu, Mar 22, 2012 at 2:34 PM, Thejas Nair<[EMAIL PROTECTED]>
>>>> wrote:
>>>>
>>>> Is this what you are looking for  ? -
>>>>>
>>>>>
>>>>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>>>>
>>>>> B = foreach A generate *, '20120322' as date;
>>>>>
>>>>> STORE B into ...
>>>>>
>>>>> Thanks,
>>>>> Thejas
>>>>>
>>>>
>>>>
>>>> this I am assuming will run another map task? I was trying to avoid
>>>> that.
>>>>
>>>>
>>>>>
>>>>>
>>>>> On 3/22/12 1:13 PM, Mohit Anchlia wrote:
>>>>>
>>>>> Yes that's exactly what I am asking. Reading from flat file and then
>>>>>> inserting it into the database. And I want to insert date before
>>>>>>
>>>>> storing.
>>>>
>>>>>
>>>>>> for eg I want to add date before A gets stored:
>>>>>>
>>>>>>
>>>>>> A = LOAD '$in' USING PigStorage('\t') AS (...
>>>>>>
>>>>>> STORE A into ...
>>>>>>
>>>>>> On Thu, Mar 22, 2012 at 12:54 PM, Jonathan Coveney<[EMAIL PROTECTED]
>>>>>>
>>>>> **
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> Do you mean you're reading a relation from Hadoop, and want to append
>>>>>>
>>>>> the
>>>>
>>>>>  date to the row before inserting it? I'm not quite sure what you're
>>>>>>> asking
>>>>>>> for.
>>>>>>>
>>>>>>>  >  >>>  2012/3/22 Mohit Anchlia<[EMAIL PROTECTED]****>
>>>
>>>>
>>>>>>> Sorry I mean to ask if there is any way to insert date into the ALIAS
>>>>>>>
>>>>>> so
>>>>
>>>>>  that I can use it before storing it into DB.
>>>>>>>>
>>>>>>>> On Thu, Mar 22, 2012 at 12:47 PM, Mohit Anchlia<
>>>>>>>>
>>>>>>> [EMAIL PROTECTED]
>>>>
>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I am reading bunch of columns from a flat file and inserting it into
>>>>>>>>
>>>>>>>>>
>>>>>>>>> the
>>>>>>>>
>>>>>>>
>>>>>>> database. Is there a way to also insert date?
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>