Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Passing schema inside Load functionc


+
praveenesh kumar 2012-02-03, 12:35
+
Stan Rosenberg 2012-02-03, 22:32
+
praveenesh kumar 2012-02-03, 22:35
Copy link to this message
-
Re: Passing schema inside Load functionc
Stan Rosenberg 2012-02-03, 22:42
Hi Praveenesh,

Assuming you have already read these:

http://ofps.oreilly.com/titles/9781449302641/load_and_store_funcs.html
http://pig.apache.org/docs/r0.9.2/udf.html#load-store-functions

my next step would be to peruse the source code of some existing
loaders, e.g., PigStorage.

Best,

stan
On Fri, Feb 3, 2012 at 5:35 PM, praveenesh kumar <[EMAIL PROTECTED]> wrote:
> Thanks Stan,
> If you were facing this kind of scenario, how would you have proceeded ?
> Can you give me some pointers on how to write custom loader, some good
> tutorials..on it
> What is the current practice in order to solve the above scenario in pig ?
>
> Praveenesh
>
>
> On Sat, Feb 4, 2012 at 4:02 AM, Stan Rosenberg <
> [EMAIL PROTECTED]> wrote:
>
>> My hunch is you'll have to write a custom loader, but I'll let the
>> experts chime in.  E.g., AvroStorage loader can parse the schema
>> from a json file passed to it via the constructor.  I don't think
>> PigStorage has the same option.
>>
>> stan
>>
>> On Fri, Feb 3, 2012 at 7:35 AM, praveenesh kumar <[EMAIL PROTECTED]>
>> wrote:
>> > Hey guys,
>> >
>> > I am new to Pig.
>> > I was wondering is it possible to pass schema in pig load statement while
>> > loading it first time.
>> >
>> > Suppose if I have a huge dataset.. containing around 100 cols.. Is there
>> a
>> > way through which I can pass the schema defined in some other file (some
>> > kind of meta file) into pig load statement or do I have to define it
>> every
>> > time inside LOAD statement ?
>> >
>> > Thanks,
>> > Praveenesh
>>
+
praveenesh kumar 2012-02-03, 22:45
+
Stan Rosenberg 2012-02-04, 02:40
+
praveenesh kumar 2012-02-04, 06:48
+
Dmitriy Ryaboy 2012-02-06, 06:25
+
praveenesh kumar 2012-02-06, 06:35
+
Dmitriy Ryaboy 2012-02-06, 06:48
+
praveenesh kumar 2012-02-06, 06:59
+
Dmitriy Ryaboy 2012-02-06, 09:11
+
praveenesh kumar 2012-02-06, 09:17
+
Dmitriy Ryaboy 2012-02-06, 21:20