Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Stopping after load with no tuples?


+
John Farrelly 2013-04-09, 13:28
Copy link to this message
-
Re: Stopping after load with no tuples?
Hi

I haven't heard of such functionality in Pig, but it seems that you are
looking for a functionality that is not what Pig is for. It lacks some
control things like IF ELSE statements, because it is not a general-purpose
language. There are a couple of ways you can do such things:
1) Launch Pig with a bash wrapper.
2) Embed Pig into a Java or Python, etc (just like you would embed SQL into
a regular language). Pig supports it out of the box
3) Use Oozie or something similar for the jobs orchestration.

Hope that helps

Ruslan Al-Fakikh

On Tue, Apr 9, 2013 at 5:28 PM, John Farrelly <
[EMAIL PROTECTED]> wrote:

> Hi there,
>
> I have a pig script similar to the  one below.  When testing this on a
> cluster with an empty file, I see it taking ages to complete - It goes
> through all the commands and runs the jobs across the tracker.  In our test
> environment, we will (for various reasons) have an input directory with a
> file which is empty.  Is there any way to tell pig to exit the script if
> there is no data available (or a command like EXIT IF data EMPTY?).  My UDF
> loader does return null as the tuple when it reads an empty file, so
> ideally I'd like it to stop processing.  Is there any way to do this?
>
> Thanks,
> John.
>
> define MyDataLoader com.example.test.MyDataLoader('config1');
> raw = LOAD '$inputdir' USING MyDataLoader AS (date:chararray,
> values:bag{t:(policy:chararray)});
>
> data1    = FOREACH raw GENERATE date, .....
> data2    = GROUP data1 BY (date, .....
> data3    = FOREACH data3 GENERATE group.date, .....
>
> STORE data3 INTO 'MyDataStore'
> </pre>****************************************************************************************<br>This
> email and any files transmitted with are confidential and intended solely
> for the<br>use of the individual or entity to whom they are addressed.  If
> you have received this<br>email in error then please delete it and notify
> the sender. Do not make a copy or forward<br>it to anyone.  This footnote
> also confirms that this email message has been swept for the<br>presence of
> computer viruses.<br><br>Adaptive Mobile Security Ltd, Ferry House, 48
> Lower Mount Street, Dublin 2, Ireland<br>Directors: B. Collins, G.
> Maclachlan (UK), N. Grierson (UK), J. Ennis (UK), D. Summers
> (UK).<br>Registered in Ireland, Company No. 370343, VAT
> Reg.No.IE6390343O<br>****************************************************************************************</pre>
>
>
+
John Farrelly 2013-04-18, 12:57