Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - STORE with variable?


Copy link to this message
-
STORE with variable?
Dexin Wang 2011-03-08, 20:29
Is there a way to use STORE with variable or some other way to achieve what
I need.

I have something like this:

grunt> DESCRIBE A;
A: {f1, f2, f3, ...}

grunt> DUMP A;
(v1, x2, x3, ...)
(v2, x4, x5, ...)
(v1, x6, x6, ...)
...

I do so processing and then group by f1 and would like to save the result in
different directories for different f1, like this:

     /result/f1/result_for_v1
     /result/f2/result_for_v2
     /result/f2/result_for_v2
     ...

I know I could use SPLIT, but I have 100+ unique values for f1, and number
of uniques varies each time I process. It will be nice I don't have list 100
BY lines with SPLIT and I certainly do not want to maintain the list of
possible values for f1 in my Pig script.

Thanks!
Dexin