Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Run a job async


+
Prashant Kommireddi 2013-01-24, 00:48
+
Jonathan Coveney 2013-01-24, 01:44
+
Prashant Kommireddi 2013-01-24, 02:04
+
Jonathan Coveney 2013-01-24, 05:09
+
Prashant Kommireddi 2013-01-24, 05:22
+
Alan Gates 2013-01-24, 16:37
+
Prashant Kommireddi 2013-01-24, 17:42
+
Alan Gates 2013-01-24, 17:46
Copy link to this message
-
Re: Run a job async
Prashant Kommireddi 2013-02-06, 00:30
Thanks Alan, this was helpful.

On Thu, Jan 24, 2013 at 9:46 AM, Alan Gates <[EMAIL PROTECTED]> wrote:

> You might want to look at webhcat's code.  It produces a servlet that it
> embeds in a jetty server.  You may be able to copy paste this to get what
> you want.
>
> The code of interest is in the hcat repository under webhcat/svr.
>
> Alan.
>
> On Jan 24, 2013, at 9:42 AM, Prashant Kommireddi wrote:
>
> > Thanks Alan. We are trying to plug Pig into our existing app server.
> > We have already done this for Java MR. The difficulty we are facing is
> > with the fact that we can use JobClient.submitJob and jobtracker's job
> > end notification to run jobs async, whereas PigServer.executeBatch
> > blocks until pig job is complete.
> >
> > Sent from my iPhone
> >
> > On Jan 24, 2013, at 9:31 AM, Alan Gates <[EMAIL PROTECTED]> wrote:
> >
> >> If you're looking for an app server for Pig I'd take a look at a couple
> of other projects already out there that can do this:
> >>
> >> 1) webhcat (fka Templeton, now part of the HCatalog project).  It
> provides a REST API that launches Pig, Hive, or MR jobs and allows you to
> manage them, get results, etc.  It's in HCatalog 0.5, which is in the
> release candidate state.  You can go to
> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/and pick up the release candidate.
> >>
> >> 2) Oozie.  Oozie's a workflow engine for Hadoop, but it also supports
> submission of single Pig or MR jobs via REST.  It may be a little
> heavyweight for what you want but it works.
> >>
> >> Alan.
> >>
> >> On Jan 23, 2013, at 9:22 PM, Prashant Kommireddi wrote:
> >>
> >>> Both. Think of it as an app server handling all of these requests.
> >>>
> >>> Sent from my iPhone
> >>>
> >>> On Jan 23, 2013, at 9:09 PM, Jonathan Coveney <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>>> Thousands of requests, or thousands of Pig jobs? Or both?
> >>>>
> >>>>
> >>>> 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> >>>>
> >>>>> Did not want to have several threads launched for this. We might have
> >>>>> thousands of requests coming in, and the app is doing a lot more
> than only
> >>>>> Pig.
> >>>>>
> >>>>> On Wed, Jan 23, 2013 at 5:44 PM, Jonathan Coveney <
> [EMAIL PROTECTED]
> >>>>>> wrote:
> >>>>>
> >>>>>> start a separate Process which runs Pig?
> >>>>>>
> >>>>>>
> >>>>>> 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> >>>>>>
> >>>>>>> Hey guys,
> >>>>>>>
> >>>>>>> I am trying to do the following:
> >>>>>>>
> >>>>>>> 1. Launch a pig job asynchronously via Java program
> >>>>>>> 2. Get a notification once the job is complete (something similar
> to
> >>>>>>> Hadoop callback with a servlet)
> >>>>>>>
> >>>>>>> I looked at PigServer.executeBatch() and it seems to be waiting
> until
> >>>>> job
> >>>>>>> completes.This is not what I would like my app to do.
> >>>>>>>
> >>>>>>> Any ideas?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>
> >>>>>
> >>
>
>
+
Jonathan Coveney 2013-01-24, 06:56
+
Praveen M 2013-01-24, 15:02
+
Ramakrishna Nalam 2013-01-25, 03:57
+
Jonathan Coveney 2013-01-25, 04:39
+
Ramakrishna Nalam 2013-01-25, 07:18
+
Cheolsoo Park 2013-01-25, 17:08
+
Jonathan Coveney 2013-01-25, 17:37
+
Rohini Palaniswamy 2013-01-26, 00:23
+
Cheolsoo Park 2013-01-26, 02:55
+
Bill Graham 2013-01-24, 01:35