Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Run a job async


+
Prashant Kommireddi 2013-01-24, 00:48
+
Jonathan Coveney 2013-01-24, 01:44
+
Prashant Kommireddi 2013-01-24, 02:04
+
Jonathan Coveney 2013-01-24, 05:09
+
Prashant Kommireddi 2013-01-24, 05:22
+
Alan Gates 2013-01-24, 16:37
Copy link to this message
-
Re: Run a job async
Prashant Kommireddi 2013-01-24, 17:42
Thanks Alan. We are trying to plug Pig into our existing app server.
We have already done this for Java MR. The difficulty we are facing is
with the fact that we can use JobClient.submitJob and jobtracker's job
end notification to run jobs async, whereas PigServer.executeBatch
blocks until pig job is complete.

Sent from my iPhone

On Jan 24, 2013, at 9:31 AM, Alan Gates <[EMAIL PROTECTED]> wrote:

> If you're looking for an app server for Pig I'd take a look at a couple of other projects already out there that can do this:
>
> 1) webhcat (fka Templeton, now part of the HCatalog project).  It provides a REST API that launches Pig, Hive, or MR jobs and allows you to manage them, get results, etc.  It's in HCatalog 0.5, which is in the release candidate state.  You can go to http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/ and pick up the release candidate.
>
> 2) Oozie.  Oozie's a workflow engine for Hadoop, but it also supports submission of single Pig or MR jobs via REST.  It may be a little heavyweight for what you want but it works.
>
> Alan.
>
> On Jan 23, 2013, at 9:22 PM, Prashant Kommireddi wrote:
>
>> Both. Think of it as an app server handling all of these requests.
>>
>> Sent from my iPhone
>>
>> On Jan 23, 2013, at 9:09 PM, Jonathan Coveney <[EMAIL PROTECTED]> wrote:
>>
>>> Thousands of requests, or thousands of Pig jobs? Or both?
>>>
>>>
>>> 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
>>>
>>>> Did not want to have several threads launched for this. We might have
>>>> thousands of requests coming in, and the app is doing a lot more than only
>>>> Pig.
>>>>
>>>> On Wed, Jan 23, 2013 at 5:44 PM, Jonathan Coveney <[EMAIL PROTECTED]
>>>>> wrote:
>>>>
>>>>> start a separate Process which runs Pig?
>>>>>
>>>>>
>>>>> 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
>>>>>
>>>>>> Hey guys,
>>>>>>
>>>>>> I am trying to do the following:
>>>>>>
>>>>>> 1. Launch a pig job asynchronously via Java program
>>>>>> 2. Get a notification once the job is complete (something similar to
>>>>>> Hadoop callback with a servlet)
>>>>>>
>>>>>> I looked at PigServer.executeBatch() and it seems to be waiting until
>>>> job
>>>>>> completes.This is not what I would like my app to do.
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>
>>>>
>
+
Alan Gates 2013-01-24, 17:46
+
Prashant Kommireddi 2013-02-06, 00:30
+
Jonathan Coveney 2013-01-24, 06:56
+
Praveen M 2013-01-24, 15:02
+
Ramakrishna Nalam 2013-01-25, 03:57
+
Jonathan Coveney 2013-01-25, 04:39
+
Ramakrishna Nalam 2013-01-25, 07:18
+
Cheolsoo Park 2013-01-25, 17:08
+
Jonathan Coveney 2013-01-25, 17:37
+
Rohini Palaniswamy 2013-01-26, 00:23
+
Cheolsoo Park 2013-01-26, 02:55
+
Bill Graham 2013-01-24, 01:35