Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Injecting plans


Copy link to this message
-
Re: Injecting plans
Thanks Bill. I think the issue would be with the fact that I am trying to
inject the property (base path) from an external framework into
Load/StoreFunc. An analogy could be a fancy random generator that resides
in a framework has to be used to setLocation. However, "random generator
logic" can not be placed within a Load/StoreFunc. So a user that uses, say
CustomPigStorage passes his script to the framework and the framework now
needs to alter the i/o locations.
   1. User submits script -> Framework.getService().submitPigScript(File
   script);
   2. Script is using a Load/StoreFunc
   3. Framework has to parse the script to determine the user provided
   locations?

I understand when you say it could be done dynamically within Funcs, but I
guess I am not very clear on how we could alter i/o locations from outside
of Pig.
Pardon me if I am not explaining it well enough!

Julien, this is not for testing or for injecting data.
On Mon, Feb 11, 2013 at 7:47 PM, Bill Graham <[EMAIL PROTECTED]> wrote:

> You can do it dynamically in the StoreFunc/LoadFunc, but not before the LP
> is built (since the LP really isn't yet involved with the physical
> location). Why would you need to change the location that early in the
> process?
>
>
> On Mon, Feb 11, 2013 at 6:30 PM, Julien Le Dem <[EMAIL PROTECTED]> wrote:
>
> > Is it for testing or something totally different?
> > you can also look at o.a.p.builtin.mock.Storage for injecting data
> > Julien
> >
> >
> > On Mon, Feb 11, 2013 at 6:17 PM, Prashant Kommireddi <
> [EMAIL PROTECTED]>wrote:
> >
> >> That would work in the case when it's a static path? The problem here is
> >> my
> >> base path will vary entirely for each different Pig job, and I would not
> >> like the client to set that. Rather my framework plugs-in the base path
> >> and
> >> I am guessing that can happen only after parsing/buildLP?
> >>
> >> On Mon, Feb 11, 2013 at 5:55 PM, Bill Graham <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >> > We've done this before by overriding  relativeToAbsolutePath
> >> > and setLocation in the LoadFunc, or the correspoding methods in
> >> StoreFunc.
> >> >
> >> > On Mon, Feb 11, 2013 at 2:10 PM, Prashant Kommireddi <
> >> [EMAIL PROTECTED]
> >> > >wrote:
> >> >
> >> > > Hey,
> >> > >
> >> > > I wanted to run an idea by you guys. I have a use-case where I try
> >> > > injecting load/store paths into the script. So if a user says A > load
> >> > > 'input'; I would like to add a base path to it and make it A = load
> >> > > 'base_path/foo/bar/input'.
> >> > >
> >> > > I would like to achieve this programatically, and 1 way I can of
> >> doing it
> >> > > is by allowing setter on LOLoad/LOStore that can modify the
> FileSpec.
> >> Is
> >> > > there a cleaner or better way to approach this?
> >> > >
> >> > > In the future I would probably be meddling with more than just
> >> Load/Store
> >> > > (for eg, disabling rmf or mkdir commands etc) but this is something
> I
> >> am
> >> > > looking at currently.
> >> > >
> >> > > Thanks,
> >> > > Prashant
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > *Note that I'm no longer using my Yahoo! email address. Please email
> me
> >> at
> >> > [EMAIL PROTECTED] going forward.*
> >> >
> >>
> >
> >
>
>
> --
> *Note that I'm no longer using my Yahoo! email address. Please email me at
> [EMAIL PROTECTED] going forward.*
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB