Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Run a job async


+
Prashant Kommireddi 2013-01-24, 00:48
+
Jonathan Coveney 2013-01-24, 01:44
+
Prashant Kommireddi 2013-01-24, 02:04
+
Jonathan Coveney 2013-01-24, 05:09
+
Prashant Kommireddi 2013-01-24, 05:22
+
Alan Gates 2013-01-24, 16:37
+
Prashant Kommireddi 2013-01-24, 17:42
+
Alan Gates 2013-01-24, 17:46
+
Prashant Kommireddi 2013-02-06, 00:30
+
Jonathan Coveney 2013-01-24, 06:56
+
Praveen M 2013-01-24, 15:02
+
Ramakrishna Nalam 2013-01-25, 03:57
+
Jonathan Coveney 2013-01-25, 04:39
Copy link to this message
-
Re: Run a job async
That clarifies it for me, thanks a lot.

Regards,
Rama.
On Fri, Jan 25, 2013 at 10:09 AM, Jonathan Coveney <[EMAIL PROTECTED]>wrote:

> Well, when I say that Pig is not multi-threaded, what I mean is that if you
> have multiple threads that run a query via PigServer, there is a great
> chance of the internals clashing because of the use of static variables
> within Pig. Pig itself, when running a single query, is multi-threaded.
> It's just not "multi-threaded" in the sense that multiple instances can
> safely be run in the same JVM.
>
>
> 2013/1/24 Ramakrishna Nalam <[EMAIL PROTECTED]>
>
> > Hi Jonathan,
> >
> > Pardon if it's a naive question, but Interesting that you say Pig is not
> > multithreaded.
> > We're using Pig 0.10.0, and looking at the code, it seems to do the right
> > things to handle multi threaded requests (ThreadLocal for ScriptState for
> > eg).
> >
> > Would be great if you can point out to the kind of issues there could be.
> >
> >
> > Regards,
> > Rama.
> >
> >
> >
> > On Thu, Jan 24, 2013 at 8:32 PM, Praveen M <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Are there any plans on making the pigserver multi-threaded?
> > >
> > > since there is "PigProcessNotificationListener" to subscribe for async
> > > callbacks when the pig job completes, is there any real need to keep
> the
> > > pig job submitting thread waiting until the job completes?
> > >
> > > Is this just a shortcoming today or are there more concrete reasons
> > against
> > > providing with a pigserver which can submit to the cluster in mapreduce
> > > mode async?
> > >
> > > Thanks,
> > > Praveen
> > >
> > >
> > >
> > > On Wed, Jan 23, 2013 at 10:56 PM, Jonathan Coveney <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > I think whatever way you slice it, handling thousands of pig jobs
> > > > asynchronously is going to be a bear. I mean, this is essentially
> what
> > > the
> > > > job tracker does, albeit with a lot less information.
> > > >
> > > > Either way, Pig is not multi-threaded so having more than one
> instance
> > of
> > > > Pig in the same JVM is going to start causing problems (which is
> why, I
> > > > imagine, there is no async way to call Pig). So multiple processes is
> > > > really the only way around it that I know of.
> > > >
> > > > At Twitter we have a deployment of mesos, and our long term solution
> is
> > > > going to be running all of our pig jobs on mesos, in the short term
> by
> > > > deploying daemons that run pig jobs as local processes.
> > > >
> > > >
> > > > 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> > > >
> > > > > Both. Think of it as an app server handling all of these requests.
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > On Jan 23, 2013, at 9:09 PM, Jonathan Coveney <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > > > Thousands of requests, or thousands of Pig jobs? Or both?
> > > > > >
> > > > > >
> > > > > > 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> > > > > >
> > > > > >> Did not want to have several threads launched for this. We might
> > > have
> > > > > >> thousands of requests coming in, and the app is doing a lot more
> > > than
> > > > > only
> > > > > >> Pig.
> > > > > >>
> > > > > >> On Wed, Jan 23, 2013 at 5:44 PM, Jonathan Coveney <
> > > [EMAIL PROTECTED]
> > > > > >>> wrote:
> > > > > >>
> > > > > >>> start a separate Process which runs Pig?
> > > > > >>>
> > > > > >>>
> > > > > >>> 2013/1/23 Prashant Kommireddi <[EMAIL PROTECTED]>
> > > > > >>>
> > > > > >>>> Hey guys,
> > > > > >>>>
> > > > > >>>> I am trying to do the following:
> > > > > >>>>
> > > > > >>>>   1. Launch a pig job asynchronously via Java program
> > > > > >>>>   2. Get a notification once the job is complete (something
> > > similar
> > > > to
> > > > > >>>>   Hadoop callback with a servlet)
> > > > > >>>>
> > > > > >>>> I looked at PigServer.executeBatch() and it seems to be
> waiting
> > > > until
> > > > > >> job
> > > > > >>>> completes.This is not what I would like my app to do.
+
Cheolsoo Park 2013-01-25, 17:08
+
Jonathan Coveney 2013-01-25, 17:37
+
Rohini Palaniswamy 2013-01-26, 00:23
+
Cheolsoo Park 2013-01-26, 02:55
+
Bill Graham 2013-01-24, 01:35
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB