Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Passing schema inside Load functionc


+
praveenesh kumar 2012-02-03, 12:35
+
Stan Rosenberg 2012-02-03, 22:32
+
praveenesh kumar 2012-02-03, 22:35
Copy link to this message
-
Re: Passing schema inside Load functionc
Hi Praveenesh,

Assuming you have already read these:

http://ofps.oreilly.com/titles/9781449302641/load_and_store_funcs.html
http://pig.apache.org/docs/r0.9.2/udf.html#load-store-functions

my next step would be to peruse the source code of some existing
loaders, e.g., PigStorage.

Best,

stan
On Fri, Feb 3, 2012 at 5:35 PM, praveenesh kumar <[EMAIL PROTECTED]> wrote:
> Thanks Stan,
> If you were facing this kind of scenario, how would you have proceeded ?
> Can you give me some pointers on how to write custom loader, some good
> tutorials..on it
> What is the current practice in order to solve the above scenario in pig ?
>
> Praveenesh
>
>
> On Sat, Feb 4, 2012 at 4:02 AM, Stan Rosenberg <
> [EMAIL PROTECTED]> wrote:
>
>> My hunch is you'll have to write a custom loader, but I'll let the
>> experts chime in.  E.g., AvroStorage loader can parse the schema
>> from a json file passed to it via the constructor.  I don't think
>> PigStorage has the same option.
>>
>> stan
>>
>> On Fri, Feb 3, 2012 at 7:35 AM, praveenesh kumar <[EMAIL PROTECTED]>
>> wrote:
>> > Hey guys,
>> >
>> > I am new to Pig.
>> > I was wondering is it possible to pass schema in pig load statement while
>> > loading it first time.
>> >
>> > Suppose if I have a huge dataset.. containing around 100 cols.. Is there
>> a
>> > way through which I can pass the schema defined in some other file (some
>> > kind of meta file) into pig load statement or do I have to define it
>> every
>> > time inside LOAD statement ?
>> >
>> > Thanks,
>> > Praveenesh
>>
+
praveenesh kumar 2012-02-03, 22:45
+
Stan Rosenberg 2012-02-04, 02:40
+
praveenesh kumar 2012-02-04, 06:48
+
Dmitriy Ryaboy 2012-02-06, 06:25
+
praveenesh kumar 2012-02-06, 06:35
+
Dmitriy Ryaboy 2012-02-06, 06:48
+
praveenesh kumar 2012-02-06, 06:59
+
Dmitriy Ryaboy 2012-02-06, 09:11
+
praveenesh kumar 2012-02-06, 09:17
+
Dmitriy Ryaboy 2012-02-06, 21:20
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB