Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Stopping after load with no tuples?


+
John Farrelly 2013-04-09, 13:28
Copy link to this message
-
Re: Stopping after load with no tuples?
Hi

I haven't heard of such functionality in Pig, but it seems that you are
looking for a functionality that is not what Pig is for. It lacks some
control things like IF ELSE statements, because it is not a general-purpose
language. There are a couple of ways you can do such things:
1) Launch Pig with a bash wrapper.
2) Embed Pig into a Java or Python, etc (just like you would embed SQL into
a regular language). Pig supports it out of the box
3) Use Oozie or something similar for the jobs orchestration.

Hope that helps

Ruslan Al-Fakikh

On Tue, Apr 9, 2013 at 5:28 PM, John Farrelly <
[EMAIL PROTECTED]> wrote:

> Hi there,
>
> I have a pig script similar to the  one below.  When testing this on a
> cluster with an empty file, I see it taking ages to complete - It goes
> through all the commands and runs the jobs across the tracker.  In our test
> environment, we will (for various reasons) have an input directory with a
> file which is empty.  Is there any way to tell pig to exit the script if
> there is no data available (or a command like EXIT IF data EMPTY?).  My UDF
> loader does return null as the tuple when it reads an empty file, so
> ideally I'd like it to stop processing.  Is there any way to do this?
>
> Thanks,
> John.
>
> define MyDataLoader com.example.test.MyDataLoader('config1');
> raw = LOAD '$inputdir' USING MyDataLoader AS (date:chararray,
> values:bag{t:(policy:chararray)});
>
> data1    = FOREACH raw GENERATE date, .....
> data2    = GROUP data1 BY (date, .....
> data3    = FOREACH data3 GENERATE group.date, .....
>
> STORE data3 INTO 'MyDataStore'
> </pre>****************************************************************************************<br>This
> email and any files transmitted with are confidential and intended solely
> for the<br>use of the individual or entity to whom they are addressed.  If
> you have received this<br>email in error then please delete it and notify
> the sender. Do not make a copy or forward<br>it to anyone.  This footnote
> also confirms that this email message has been swept for the<br>presence of
> computer viruses.<br><br>Adaptive Mobile Security Ltd, Ferry House, 48
> Lower Mount Street, Dublin 2, Ireland<br>Directors: B. Collins, G.
> Maclachlan (UK), N. Grierson (UK), J. Ennis (UK), D. Summers
> (UK).<br>Registered in Ireland, Company No. 370343, VAT
> Reg.No.IE6390343O<br>****************************************************************************************</pre>
>
>
+
John Farrelly 2013-04-18, 12:57
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB