Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - access schema defined in LOAD statement in custom LoadFunc?


+
Jim Donofrio 2012-09-15, 05:13
+
Alan Gates 2012-09-15, 16:38
+
Jim Donofrio 2012-09-17, 03:11
Copy link to this message
-
Re: access schema defined in LOAD statement in custom LoadFunc?
Dmitriy Ryaboy 2012-09-17, 05:06
I am not sure why pushProjection doesn't solve your dilemma?
This is what we use in HBaseStorage, and ElephantBird uses in thrift
and protobuf loaders.

D

On Sun, Sep 16, 2012 at 8:11 PM, Jim Donofrio <[EMAIL PROTECTED]> wrote:
> I guess a workaround could be to Base64 decode the pig.script property and
> look for AS statements
>
>
> On 09/15/2012 12:38 PM, Alan Gates wrote:
>>
>> Unfortunately, no.  I agree we should add that to the LoadFunc interface.
>>
>> Alan.
>>
>> On Sep 15, 2012, at 1:13 AM, Jim Donofrio wrote:
>>
>>> Is there anyway within a LoadFunc to access the schema that a user
>>> defines after AS in a LOAD statement? Is there some property I can access in
>>> the UDFContext or ? pushProjection provides the schema from a FOREACH
>>> GENERATE and getSchema seems to be only meant to read part of the actual
>>> file or a JSON schema file off disk.
>>
>>
>