Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Run a job async


+
Prashant Kommireddi 2013-01-24, 00:48
+
Jonathan Coveney 2013-01-24, 01:44
+
Prashant Kommireddi 2013-01-24, 02:04
+
Jonathan Coveney 2013-01-24, 05:09
+
Prashant Kommireddi 2013-01-24, 05:22
+
Alan Gates 2013-01-24, 16:37
+
Prashant Kommireddi 2013-01-24, 17:42
+
Alan Gates 2013-01-24, 17:46
+
Prashant Kommireddi 2013-02-06, 00:30
+
Jonathan Coveney 2013-01-24, 06:56
+
Praveen M 2013-01-24, 15:02
+
Ramakrishna Nalam 2013-01-25, 03:57
Copy link to this message
-
Re: Run a job async
Well, when I say that Pig is not multi-threaded, what I mean is that if you
have multiple threads that run a query via PigServer, there is a great
chance of the internals clashing because of the use of static variables
within Pig. Pig itself, when running a single query, is multi-threaded.
It's just not "multi-threaded" in the sense that multiple instances can
safely be run in the same JVM.
2013/1/24 Ramakrishna Nalam <[EMAIL PROTECTED]>

> Hi Jonathan,
>
> Pardon if it's a naive question, but Interesting that you say Pig is not
> multithreaded.
> We're using Pig 0.10.0, and looking at the code, it seems to do the right
> things to handle multi threaded requests (ThreadLocal for ScriptState for
> eg).
>
> Would be great if you can point out to the kind of issues there could be.
>
>
> Regards,
> Rama.
>
>
>
> On Thu, Jan 24, 2013 at 8:32 PM, Praveen M <[EMAIL PROTECTED]>
> wrote:
>
> > Are there any plans on making the pigserver multi-threaded?
> >
> > since there is "PigProcessNotificationListener" to subscribe for async
> > callbacks when the pig job completes, is there any real need to keep the
> > pig job submitting thread waiting until the job completes?
> >
> > Is this just a shortcoming today or are there more concrete reasons
> against
> > providing with a pigserver which can submit to the cluster in mapreduce
> > mode async?
> >
> > Thanks,
> > Praveen
> >
> >
> >
> > On Wed, Jan 23, 2013 at 10:56 PM, Jonathan Coveney <[EMAIL PROTECTED]
> > >wrote:
> >
> > > I think whatever way you slice it, handling thousands of pig jobs
> > > asynchronously is going to be a bear. I mean, this is essentially what
> > the
> > > job tracker does, albeit with a lot less information.
> > >
> > > Either way, Pig is not multi-threaded so having more than one instance
> of
> > > Pig in the same JVM is going to start causing problems (which is why, I
> > > imagine, there is no async way to call Pig). So multiple processes is
> > > really the only way around it that I know of.
> > >
> > > At Twitter we have a deployment of mesos, and our long term solution is
> > > going to be running all of our pig jobs on mesos, in the short term by
> > > deploying daemons that run pig jobs as local processes.
> > >
> > >
> > > 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> > >
> > > > Both. Think of it as an app server handling all of these requests.
> > > >
> > > > Sent from my iPhone
> > > >
> > > > On Jan 23, 2013, at 9:09 PM, Jonathan Coveney <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Thousands of requests, or thousands of Pig jobs? Or both?
> > > > >
> > > > >
> > > > > 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> > > > >
> > > > >> Did not want to have several threads launched for this. We might
> > have
> > > > >> thousands of requests coming in, and the app is doing a lot more
> > than
> > > > only
> > > > >> Pig.
> > > > >>
> > > > >> On Wed, Jan 23, 2013 at 5:44 PM, Jonathan Coveney <
> > [EMAIL PROTECTED]
> > > > >>> wrote:
> > > > >>
> > > > >>> start a separate Process which runs Pig?
> > > > >>>
> > > > >>>
> > > > >>> 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> > > > >>>
> > > > >>>> Hey guys,
> > > > >>>>
> > > > >>>> I am trying to do the following:
> > > > >>>>
> > > > >>>>   1. Launch a pig job asynchronously via Java program
> > > > >>>>   2. Get a notification once the job is complete (something
> > similar
> > > to
> > > > >>>>   Hadoop callback with a servlet)
> > > > >>>>
> > > > >>>> I looked at PigServer.executeBatch() and it seems to be waiting
> > > until
> > > > >> job
> > > > >>>> completes.This is not what I would like my app to do.
> > > > >>>>
> > > > >>>> Any ideas?
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> >
> >
> > --
> > -Praveen
> >
>
+
Ramakrishna Nalam 2013-01-25, 07:18
+
Cheolsoo Park 2013-01-25, 17:08
+
Jonathan Coveney 2013-01-25, 17:37
+
Rohini Palaniswamy 2013-01-26, 00:23
+
Cheolsoo Park 2013-01-26, 02:55
+
Bill Graham 2013-01-24, 01:35