Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Run a job async


+
Prashant Kommireddi 2013-01-24, 00:48
+
Jonathan Coveney 2013-01-24, 01:44
+
Prashant Kommireddi 2013-01-24, 02:04
+
Jonathan Coveney 2013-01-24, 05:09
+
Prashant Kommireddi 2013-01-24, 05:22
+
Alan Gates 2013-01-24, 16:37
+
Prashant Kommireddi 2013-01-24, 17:42
+
Alan Gates 2013-01-24, 17:46
+
Prashant Kommireddi 2013-02-06, 00:30
+
Jonathan Coveney 2013-01-24, 06:56
+
Praveen M 2013-01-24, 15:02
+
Ramakrishna Nalam 2013-01-25, 03:57
+
Jonathan Coveney 2013-01-25, 04:39
Copy link to this message
-
Re: Run a job async
That clarifies it for me, thanks a lot.

Regards,
Rama.
On Fri, Jan 25, 2013 at 10:09 AM, Jonathan Coveney <[EMAIL PROTECTED]>wrote:

> Well, when I say that Pig is not multi-threaded, what I mean is that if you
> have multiple threads that run a query via PigServer, there is a great
> chance of the internals clashing because of the use of static variables
> within Pig. Pig itself, when running a single query, is multi-threaded.
> It's just not "multi-threaded" in the sense that multiple instances can
> safely be run in the same JVM.
>
>
> 2013/1/24 Ramakrishna Nalam <[EMAIL PROTECTED]>
>
> > Hi Jonathan,
> >
> > Pardon if it's a naive question, but Interesting that you say Pig is not
> > multithreaded.
> > We're using Pig 0.10.0, and looking at the code, it seems to do the right
> > things to handle multi threaded requests (ThreadLocal for ScriptState for
> > eg).
> >
> > Would be great if you can point out to the kind of issues there could be.
> >
> >
> > Regards,
> > Rama.
> >
> >
> >
> > On Thu, Jan 24, 2013 at 8:32 PM, Praveen M <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Are there any plans on making the pigserver multi-threaded?
> > >
> > > since there is "PigProcessNotificationListener" to subscribe for async
> > > callbacks when the pig job completes, is there any real need to keep
> the
> > > pig job submitting thread waiting until the job completes?
> > >
> > > Is this just a shortcoming today or are there more concrete reasons
> > against
> > > providing with a pigserver which can submit to the cluster in mapreduce
> > > mode async?
> > >
> > > Thanks,
> > > Praveen
> > >
> > >
> > >
> > > On Wed, Jan 23, 2013 at 10:56 PM, Jonathan Coveney <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > I think whatever way you slice it, handling thousands of pig jobs
> > > > asynchronously is going to be a bear. I mean, this is essentially
> what
> > > the
> > > > job tracker does, albeit with a lot less information.
> > > >
> > > > Either way, Pig is not multi-threaded so having more than one
> instance
> > of
> > > > Pig in the same JVM is going to start causing problems (which is
> why, I
> > > > imagine, there is no async way to call Pig). So multiple processes is
> > > > really the only way around it that I know of.
> > > >
> > > > At Twitter we have a deployment of mesos, and our long term solution
> is
> > > > going to be running all of our pig jobs on mesos, in the short term
> by
> > > > deploying daemons that run pig jobs as local processes.
> > > >
> > > >
> > > > 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> > > >
> > > > > Both. Think of it as an app server handling all of these requests.
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > On Jan 23, 2013, at 9:09 PM, Jonathan Coveney <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > > > Thousands of requests, or thousands of Pig jobs? Or both?
> > > > > >
> > > > > >
> > > > > > 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> > > > > >
> > > > > >> Did not want to have several threads launched for this. We might
> > > have
> > > > > >> thousands of requests coming in, and the app is doing a lot more
> > > than
> > > > > only
> > > > > >> Pig.
> > > > > >>
> > > > > >> On Wed, Jan 23, 2013 at 5:44 PM, Jonathan Coveney <
> > > [EMAIL PROTECTED]
> > > > > >>> wrote:
> > > > > >>
> > > > > >>> start a separate Process which runs Pig?
> > > > > >>>
> > > > > >>>
> > > > > >>> 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> > > > > >>>
> > > > > >>>> Hey guys,
> > > > > >>>>
> > > > > >>>> I am trying to do the following:
> > > > > >>>>
> > > > > >>>>   1. Launch a pig job asynchronously via Java program
> > > > > >>>>   2. Get a notification once the job is complete (something
> > > similar
> > > > to
> > > > > >>>>   Hadoop callback with a servlet)
> > > > > >>>>
> > > > > >>>> I looked at PigServer.executeBatch() and it seems to be
> waiting
> > > > until
> > > > > >> job
> > > > > >>>> completes.This is not what I would like my app to do.
+
Cheolsoo Park 2013-01-25, 17:08
+
Jonathan Coveney 2013-01-25, 17:37
+
Rohini Palaniswamy 2013-01-26, 00:23
+
Cheolsoo Park 2013-01-26, 02:55
+
Bill Graham 2013-01-24, 01:35