Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> PigUnit test with pig.additional.jars


Copy link to this message
-
PigUnit test with pig.additional.jars
Hi everybody,

I am trying to write a test for my pig script (which runs fine on the
command line) registering any jars via the pig.additional.jars property.

I am writing the test with PigUnit as follows:

[...]

Properties props = new Properties();
props.load(new FileInputStream("./testdata/adition.properties"));
if (System.getProperties().containsKey(EXEC_CLUSTER)) {
LOG.info("Using cluster mode");
pig = new PigServer(ExecType.MAPREDUCE, props);
} else {
LOG.info("Using default local mode");
pig = new PigServer(ExecType.LOCAL, props);
}

cluster = new Cluster(pig.getPigContext());
String[] params = {
"data_dir=testdata",
"log_dir=log",
"date=2012-04-21"
};

test = new PigTest("../calcKeywords.pig", params, pig, cluster);
test.override("aggregated_united_logs",
"aggregated_united_logs = ORDER (FOREACH grouped_united_logs GENERATE " +
"group.werbeplatz_id, group.banner_id, group.campaign_id,
group.network, " +
"group.keyword, SUM(united_logs.views), SUM(united_logs.clicks)) BY ASC " +
"werbeplatz_id, banner_id, campaign_id, network, keyword;");
test.assertOutput(new File("./testdata/expected.out"));

[...]

./testdata/adition.properties contains:

pig.import.search.path=/home/schwenk/dev/pig/macros
pig.additional.jars=/home/schwenk/dev/pig/lib/adition.jar:/home/schwenk/dev/pig/lib/adition-test.jar
udf.import.list=com.adition.pig.filtering.string.:com.adition.pig.io.

As far as I could see in the debugger the PigServer registers the
additional jars successfully. Nonetheless the test fails with the
following output, stating that it cannot resolve the import of my custom
LOAD function EscapedPigStorage wich is in adition.jar:
http://pastebin.com/V5q9dxkC

What should I do next?

Johannes Schwenk

--
Softwareentwickler (Reporting)
________________________________________________________

ADITION technologies AG
Schwarzwaldstraße 78b
79117 Freiburg

http://www.adition.com

T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
SUPPORT +49  / (0)1805 - ADITION

(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)

Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
UStIDNr.: DE 218 858 434

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB