Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Pig streaming and multiquery is buggy on local mode ?


Copy link to this message
-
Pig streaming and multiquery is buggy on local mode ?
I realize today a strange behavior of PIG in local mode (streaming +
multiquery).
I put here a minimal script to reproduce the problem.

Suppose an input file with multiple lines for example:
# myInput
1
2
3
1
2
3

The pig cript is :
# bug.pig
MyInput = LOAD 'myInput;

A = myInput GROUP BY $ 0;
B = FOREACH A GENERATE FLATTEN (myInput);
C = B STREAM THROUGH `cat`;

D = myInput GROUP BY $ 0;
E = FOREACH D GENERATE FLATTEN (myInput);
STREAM THROUGH E F = `cat`;

STORE C into 'output1;
STORE F into 'output2;

I run the script using the following command:
pig -x local bug.pig

We should find in output1 and output2 perfect copy of my input file ... but
this is not the case. We find only one line (the first line of the file)
output1/part cat *
output2/part cat *

For information, it seems that the script pig hadoop corresponding work
properly.
If I comment one of the two store operation, it works as expected (i think
it's because on multiquery is run).
+
Thomas Porez 2013-07-11, 14:18
+
Thomas Porez 2013-07-11, 14:48
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB