Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Using UDF to process whole record


Copy link to this message
-
Using UDF to process whole record
Dear all,

We are using thrift and elephant-bird to store our logs. And I wanted to
use some UDF to do complex processing on a single record, so I write some
pig like the following:

========================================================raw_data = load '$INPUT' using
com.twitter.elephantbird.pig.load.LzoThriftBlockPigLoader('$CLASSNAME')

A = FOREACH raw_data GENERATE com.udf.SomeUDF(raw_data);
B = LIMIT A 10;
DUMP B;
========================================================
But the pig will told me " ERROR 1000: Error during parsing. Scalars can be
only used with projections"

Is there anyway I could run a UDF on the raw_data here?
Best wishes,
Stanley Xu
+
Vitalii Tymchyshyn 2013-01-22, 11:27
+
Young Ng 2013-01-22, 07:34
+
Vitalii Tymchyshyn 2013-01-22, 11:34
+
Stanley Xu 2013-01-24, 03:56