Pig, mail # user - Re: access schema defined in LOAD statement in custom LoadFunc?

access schema defined in LOAD statement in custom LoadFunc?

Jim Donofrio 2012-09-15, 05:13

Is there anyway within a LoadFunc to access the schema that a user
defines after AS in a LOAD statement? Is there some property I can
access in the UDFContext or ? pushProjection provides the schema from a
FOREACH GENERATE and getSchema seems to be only meant to read part of
the actual file or a JSON schema file off disk.

Re: access schema defined in LOAD statement in custom LoadFunc?

Alan Gates 2012-09-15, 16:38

Unfortunately, no. I agree we should add that to the LoadFunc interface.

Alan.

On Sep 15, 2012, at 1:13 AM, Jim Donofrio wrote:

> Is there anyway within a LoadFunc to access the schema that a user defines after AS in a LOAD statement? Is there some property I can access in the UDFContext or ? pushProjection provides the schema from a FOREACH GENERATE and getSchema seems to be only meant to read part of the actual file or a JSON schema file off disk.

Re: access schema defined in LOAD statement in custom LoadFunc?

Jim Donofrio 2012-09-17, 03:11

I guess a workaround could be to Base64 decode the pig.script property
and look for AS statements

On 09/15/2012 12:38 PM, Alan Gates wrote:
> Unfortunately, no. I agree we should add that to the LoadFunc interface.
>
> Alan.
>
> On Sep 15, 2012, at 1:13 AM, Jim Donofrio wrote:
>
>> Is there anyway within a LoadFunc to access the schema that a user defines after AS in a LOAD statement? Is there some property I can access in the UDFContext or ? pushProjection provides the schema from a FOREACH GENERATE and getSchema seems to be only meant to read part of the actual file or a JSON schema file off disk.
>

Re: access schema defined in LOAD statement in custom LoadFunc?

Dmitriy Ryaboy 2012-09-17, 05:06

I am not sure why pushProjection doesn't solve your dilemma?
This is what we use in HBaseStorage, and ElephantBird uses in thrift
and protobuf loaders.

D

On Sun, Sep 16, 2012 at 8:11 PM, Jim Donofrio <[EMAIL PROTECTED]> wrote:
> I guess a workaround could be to Base64 decode the pig.script property and
> look for AS statements
>
>
> On 09/15/2012 12:38 PM, Alan Gates wrote:
>>
>> Unfortunately, no. I agree we should add that to the LoadFunc interface.
>>
>> Alan.
>>
>> On Sep 15, 2012, at 1:13 AM, Jim Donofrio wrote:
>>
>>> Is there anyway within a LoadFunc to access the schema that a user
>>> defines after AS in a LOAD statement? Is there some property I can access in
>>> the UDFContext or ? pushProjection provides the schema from a FOREACH
>>> GENERATE and getSchema seems to be only meant to read part of the actual
>>> file or a JSON schema file off disk.
>>
>>
>