Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> How to run pig batch?


Copy link to this message
-
Re: How to run pig batch?

the pig script is generated by a system which is more than 4000 lines.

The pig client compile it for a long time.

I have solved the problem by dividing the big script  into small ones and implementing a new pig client which will concurrently submit multiple pig scripts for computation.

Is there any other way?
thanks.
在 2012-2-21,上午5:02, Dmitriy Ryaboy 写道�

> Is something preventing you from calling them via a simple bash script?
>
> On Sun, Feb 19, 2012 at 10:58 PM, Haitao Yao <[EMAIL PROTECTED]> wrote:
>> hi, all
>>        I have tens of simple pig scripts to run. While there's no parameter name collision, I merged them into a large pig script which is about 4000 lines.
>>        But the merged pig script takes pig a lot of time merging and analyzing.
>>
>>        How can I quickly submit tens of scripts in parallel?
>>
>>        BTW, I do not want to start tens of pig process for memory issue.
>>        thanks very much.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB