Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Pig storage and load functions and Cache


+
Abhishek 2012-10-05, 21:51
+
Dmitriy Ryaboy 2012-10-08, 04:10
+
abhishek dodda 2012-10-16, 03:10
Copy link to this message
-
Re: Pig storage and load functions and Cache
Dmitriy Ryaboy 2012-10-18, 04:09
I am not sure I understand the question.
You are trying to decide how to store results of your computation?
Text (PigStorage, the default) is probably easiest to work with, but
there are many other options depending on what you want to do with the
data.
TextLoader is a loader not a storer.
BinStorage is internal to pig, and you shouldn't use it unless you
really know what you are doing.

None of this is relevant to how pig optimizes queries.

D

On Sun, Oct 7, 2012 at 9:10 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:
> Pig has multi-query execution optimization built-in. If you compute
> multiple relations in your script that share parent relations, those
> parent relations will be computed only once. You don't have to do
> anything to make that happen.
>
> If you prefer to handle your own caching, you would have to handle it
> yourself, of course.
>
> There is some academic work on reusing parts of previous runs of the
> same script (potentially on overlapping, but not identical datasets);
> the papers to read are:
> Nectar http://research.microsoft.com/apps/pubs/default.aspx?id=131525
> ReStore: http://vldb.org/pvldb/vol5/p586_imanelghandour_vldb2012.pdf
>
> There are a lot of papers on iterative mapreduce, I am sure if you
> start with ReStore citations and/or Google Scholar, you'll find some.
>
> None of that has yet made it into Pig yet; I believe a general compute
> caching framework would be very useful, and look forward to someone
> taking up that challenge..
>
> D
>
> On Fri, Oct 5, 2012 at 2:51 PM, Abhishek <[EMAIL PROTECTED]> wrote:
>> BinStorage()
>> PigDump()
>> PigStorage()
>> TextLoader()
>>
>> Load or storing in which of the above format.Will optimize the queries.
>>
>> Can cache be any where in pig.How can the cache be use ful in pig.
>>
>> Regards
>> Abhi