Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - How to run pig batch?


Copy link to this message
-
Re: How to run pig batch?
Haitao Yao 2012-02-21, 01:53

the pig script is generated by a system which is more than 4000 lines.

The pig client compile it for a long time.

I have solved the problem by dividing the big script  into small ones and implementing a new pig client which will concurrently submit multiple pig scripts for computation.

Is there any other way?
thanks.
在 2012-2-21,上午5:02, Dmitriy Ryaboy 写道�

> Is something preventing you from calling them via a simple bash script?
>
> On Sun, Feb 19, 2012 at 10:58 PM, Haitao Yao <[EMAIL PROTECTED]> wrote:
>> hi, all
>>        I have tens of simple pig scripts to run. While there's no parameter name collision, I merged them into a large pig script which is about 4000 lines.
>>        But the merged pig script takes pig a lot of time merging and analyzing.
>>
>>        How can I quickly submit tens of scripts in parallel?
>>
>>        BTW, I do not want to start tens of pig process for memory issue.
>>        thanks very much.