Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - Injecting plans


+
Prashant Kommireddi 2013-02-11, 22:10
+
Bill Graham 2013-02-12, 01:55
+
Prashant Kommireddi 2013-02-12, 02:17
+
Julien Le Dem 2013-02-12, 02:30
+
Bill Graham 2013-02-12, 03:47
Copy link to this message
-
Re: Injecting plans
Prashant Kommireddi 2013-02-12, 08:24
Thanks Bill. I think the issue would be with the fact that I am trying to
inject the property (base path) from an external framework into
Load/StoreFunc. An analogy could be a fancy random generator that resides
in a framework has to be used to setLocation. However, "random generator
logic" can not be placed within a Load/StoreFunc. So a user that uses, say
CustomPigStorage passes his script to the framework and the framework now
needs to alter the i/o locations.
   1. User submits script -> Framework.getService().submitPigScript(File
   script);
   2. Script is using a Load/StoreFunc
   3. Framework has to parse the script to determine the user provided
   locations?

I understand when you say it could be done dynamically within Funcs, but I
guess I am not very clear on how we could alter i/o locations from outside
of Pig.
Pardon me if I am not explaining it well enough!

Julien, this is not for testing or for injecting data.
On Mon, Feb 11, 2013 at 7:47 PM, Bill Graham <[EMAIL PROTECTED]> wrote:

> You can do it dynamically in the StoreFunc/LoadFunc, but not before the LP
> is built (since the LP really isn't yet involved with the physical
> location). Why would you need to change the location that early in the
> process?
>
>
> On Mon, Feb 11, 2013 at 6:30 PM, Julien Le Dem <[EMAIL PROTECTED]> wrote:
>
> > Is it for testing or something totally different?
> > you can also look at o.a.p.builtin.mock.Storage for injecting data
> > Julien
> >
> >
> > On Mon, Feb 11, 2013 at 6:17 PM, Prashant Kommireddi <
> [EMAIL PROTECTED]>wrote:
> >
> >> That would work in the case when it's a static path? The problem here is
> >> my
> >> base path will vary entirely for each different Pig job, and I would not
> >> like the client to set that. Rather my framework plugs-in the base path
> >> and
> >> I am guessing that can happen only after parsing/buildLP?
> >>
> >> On Mon, Feb 11, 2013 at 5:55 PM, Bill Graham <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >> > We've done this before by overriding  relativeToAbsolutePath
> >> > and setLocation in the LoadFunc, or the correspoding methods in
> >> StoreFunc.
> >> >
> >> > On Mon, Feb 11, 2013 at 2:10 PM, Prashant Kommireddi <
> >> [EMAIL PROTECTED]
> >> > >wrote:
> >> >
> >> > > Hey,
> >> > >
> >> > > I wanted to run an idea by you guys. I have a use-case where I try
> >> > > injecting load/store paths into the script. So if a user says A > load
> >> > > 'input'; I would like to add a base path to it and make it A = load
> >> > > 'base_path/foo/bar/input'.
> >> > >
> >> > > I would like to achieve this programatically, and 1 way I can of
> >> doing it
> >> > > is by allowing setter on LOLoad/LOStore that can modify the
> FileSpec.
> >> Is
> >> > > there a cleaner or better way to approach this?
> >> > >
> >> > > In the future I would probably be meddling with more than just
> >> Load/Store
> >> > > (for eg, disabling rmf or mkdir commands etc) but this is something
> I
> >> am
> >> > > looking at currently.
> >> > >
> >> > > Thanks,
> >> > > Prashant
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > *Note that I'm no longer using my Yahoo! email address. Please email
> me
> >> at
> >> > [EMAIL PROTECTED] going forward.*
> >> >
> >>
> >
> >
>
>
> --
> *Note that I'm no longer using my Yahoo! email address. Please email me at
> [EMAIL PROTECTED] going forward.*
>
+
Julien Le Dem 2013-02-12, 17:24
+
Bill Graham 2013-02-12, 17:29