Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - Injecting plans


+
Prashant Kommireddi 2013-02-11, 22:10
+
Bill Graham 2013-02-12, 01:55
+
Prashant Kommireddi 2013-02-12, 02:17
+
Julien Le Dem 2013-02-12, 02:30
+
Bill Graham 2013-02-12, 03:47
+
Prashant Kommireddi 2013-02-12, 08:24
Copy link to this message
-
Re: Injecting plans
Julien Le Dem 2013-02-12, 17:24
Then I would point the same direction as Bill: override
 relativeToAbsolutePath
That's more or less what HCatalog does: You use a loader/storer that
abstracts out where the data is. and based on the requested location you
point to the right files.
Out of curiosity: Why does your framework need to parse first and then
modify the plan ? Are you trying to cache some intermediary results? Are
you trying to change the logical or the physical plan?
Julien
On Tue, Feb 12, 2013 at 12:24 AM, Prashant Kommireddi
<[EMAIL PROTECTED]>wrote:

> Thanks Bill. I think the issue would be with the fact that I am trying to
> inject the property (base path) from an external framework into
> Load/StoreFunc. An analogy could be a fancy random generator that resides
> in a framework has to be used to setLocation. However, "random generator
> logic" can not be placed within a Load/StoreFunc. So a user that uses, say
> CustomPigStorage passes his script to the framework and the framework now
> needs to alter the i/o locations.
>
>
>    1. User submits script -> Framework.getService().submitPigScript(File
>    script);
>    2. Script is using a Load/StoreFunc
>    3. Framework has to parse the script to determine the user provided
>    locations?
>
> I understand when you say it could be done dynamically within Funcs, but I
> guess I am not very clear on how we could alter i/o locations from outside
> of Pig.
> Pardon me if I am not explaining it well enough!
>
> Julien, this is not for testing or for injecting data.
>
>
> On Mon, Feb 11, 2013 at 7:47 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> You can do it dynamically in the StoreFunc/LoadFunc, but not before the LP
>> is built (since the LP really isn't yet involved with the physical
>> location). Why would you need to change the location that early in the
>> process?
>>
>>
>> On Mon, Feb 11, 2013 at 6:30 PM, Julien Le Dem <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Is it for testing or something totally different?
>> > you can also look at o.a.p.builtin.mock.Storage for injecting data
>> > Julien
>> >
>> >
>> > On Mon, Feb 11, 2013 at 6:17 PM, Prashant Kommireddi <
>> [EMAIL PROTECTED]>wrote:
>> >
>> >> That would work in the case when it's a static path? The problem here
>> is
>> >> my
>> >> base path will vary entirely for each different Pig job, and I would
>> not
>> >> like the client to set that. Rather my framework plugs-in the base path
>> >> and
>> >> I am guessing that can happen only after parsing/buildLP?
>> >>
>> >> On Mon, Feb 11, 2013 at 5:55 PM, Bill Graham <[EMAIL PROTECTED]>
>> >> wrote:
>> >>
>> >> > We've done this before by overriding  relativeToAbsolutePath
>> >> > and setLocation in the LoadFunc, or the correspoding methods in
>> >> StoreFunc.
>> >> >
>> >> > On Mon, Feb 11, 2013 at 2:10 PM, Prashant Kommireddi <
>> >> [EMAIL PROTECTED]
>> >> > >wrote:
>> >> >
>> >> > > Hey,
>> >> > >
>> >> > > I wanted to run an idea by you guys. I have a use-case where I try
>> >> > > injecting load/store paths into the script. So if a user says A >> load
>> >> > > 'input'; I would like to add a base path to it and make it A = load
>> >> > > 'base_path/foo/bar/input'.
>> >> > >
>> >> > > I would like to achieve this programatically, and 1 way I can of
>> >> doing it
>> >> > > is by allowing setter on LOLoad/LOStore that can modify the
>> FileSpec.
>> >> Is
>> >> > > there a cleaner or better way to approach this?
>> >> > >
>> >> > > In the future I would probably be meddling with more than just
>> >> Load/Store
>> >> > > (for eg, disabling rmf or mkdir commands etc) but this is
>> something I
>> >> am
>> >> > > looking at currently.
>> >> > >
>> >> > > Thanks,
>> >> > > Prashant
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > *Note that I'm no longer using my Yahoo! email address. Please email
>> me
>> >> at
>> >> > [EMAIL PROTECTED] going forward.*
>> >> >
>> >>
>> >
>> >
>>
>>
>> --
>> *Note that I'm no longer using my Yahoo! email address. Please email me at
>> [EMAIL PROTECTED] going forward.*
+
Bill Graham 2013-02-12, 17:29