-Re: Order By Sampling
Thejas M Nair 2011-05-06, 19:33
The sampling algorithm for order-by samples 100 records from every map task, using a reservoir sampling algorithm.
I can't think of a way to store data that could adversely affect this sampling.
This is the class (a pig load function) that is involved in sampling - org.apache.pig.impl.builtin.RandomSampleLoader
There is room for improvement in the logic that determines the total number of samples that are taken (instead of the arbitrary 100 records/map-task). But this seems to work well for most cases.
On 5/4/11 4:08 PM, "Brock Noland" <[EMAIL PROTECTED]> wrote:
I am curious as to how PIG implements sampling for order by:
Are there things I could when storing my data which would adversely
affect this sampling?