Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Run a job async


Copy link to this message
-
Re: Run a job async
I think whatever way you slice it, handling thousands of pig jobs
asynchronously is going to be a bear. I mean, this is essentially what the
job tracker does, albeit with a lot less information.

Either way, Pig is not multi-threaded so having more than one instance of
Pig in the same JVM is going to start causing problems (which is why, I
imagine, there is no async way to call Pig). So multiple processes is
really the only way around it that I know of.

At Twitter we have a deployment of mesos, and our long term solution is
going to be running all of our pig jobs on mesos, in the short term by
deploying daemons that run pig jobs as local processes.
2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>

> Both. Think of it as an app server handling all of these requests.
>
> Sent from my iPhone
>
> On Jan 23, 2013, at 9:09 PM, Jonathan Coveney <[EMAIL PROTECTED]> wrote:
>
> > Thousands of requests, or thousands of Pig jobs? Or both?
> >
> >
> > 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> >
> >> Did not want to have several threads launched for this. We might have
> >> thousands of requests coming in, and the app is doing a lot more than
> only
> >> Pig.
> >>
> >> On Wed, Jan 23, 2013 at 5:44 PM, Jonathan Coveney <[EMAIL PROTECTED]
> >>> wrote:
> >>
> >>> start a separate Process which runs Pig?
> >>>
> >>>
> >>> 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> >>>
> >>>> Hey guys,
> >>>>
> >>>> I am trying to do the following:
> >>>>
> >>>>   1. Launch a pig job asynchronously via Java program
> >>>>   2. Get a notification once the job is complete (something similar to
> >>>>   Hadoop callback with a servlet)
> >>>>
> >>>> I looked at PigServer.executeBatch() and it seems to be waiting until
> >> job
> >>>> completes.This is not what I would like my app to do.
> >>>>
> >>>> Any ideas?
> >>>>
> >>>> Thanks,
> >>>>
> >>>
> >>
>