Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Injecting plans


Copy link to this message
-
Re: Injecting plans
Julien Le Dem 2013-02-12, 02:30
Is it for testing or something totally different?
you can also look at o.a.p.builtin.mock.Storage for injecting data
Julien
On Mon, Feb 11, 2013 at 6:17 PM, Prashant Kommireddi <[EMAIL PROTECTED]>wrote:

> That would work in the case when it's a static path? The problem here is my
> base path will vary entirely for each different Pig job, and I would not
> like the client to set that. Rather my framework plugs-in the base path and
> I am guessing that can happen only after parsing/buildLP?
>
> On Mon, Feb 11, 2013 at 5:55 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
> > We've done this before by overriding  relativeToAbsolutePath
> > and setLocation in the LoadFunc, or the correspoding methods in
> StoreFunc.
> >
> > On Mon, Feb 11, 2013 at 2:10 PM, Prashant Kommireddi <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > Hey,
> > >
> > > I wanted to run an idea by you guys. I have a use-case where I try
> > > injecting load/store paths into the script. So if a user says A = load
> > > 'input'; I would like to add a base path to it and make it A = load
> > > 'base_path/foo/bar/input'.
> > >
> > > I would like to achieve this programatically, and 1 way I can of doing
> it
> > > is by allowing setter on LOLoad/LOStore that can modify the FileSpec.
> Is
> > > there a cleaner or better way to approach this?
> > >
> > > In the future I would probably be meddling with more than just
> Load/Store
> > > (for eg, disabling rmf or mkdir commands etc) but this is something I
> am
> > > looking at currently.
> > >
> > > Thanks,
> > > Prashant
> > >
> >
> >
> >
> > --
> > *Note that I'm no longer using my Yahoo! email address. Please email me
> at
> > [EMAIL PROTECTED] going forward.*
> >
>