-Re: Early projection and lazy casting
Jie Li 2011-12-03, 00:45
Why do joins prevent the early projection? Actually join has the greatest
need for it.
On Fri, Dec 2, 2011 at 7:33 PM, Jonathan Coveney <[EMAIL PROTECTED]> wrote:
> In what context? I always thought that it generally could, but that if you
> do joins it doesn't. Would be curious to know more from someone who
> 2011/12/2 Jie Li <[EMAIL PROTECTED]>
> > Hi all,
> > We just figured out Pig 0.9.1 doesn't drop those non-necessary fields
> > which really affects the performance. Though
> > that "As part of its optimizations Pig analyzes Pig Latin scripts and
> > determines what fields in an input it needs at each step in the script.
> > uses this information to aggressively drop fields it no longer needs."
> > We also found that Pig casts the data into the types defined in the
> > which is usually unnecessary, as most of them will be soon dropped.
> > To work around these, we have to manually drop those fields and remove
> > types in the schema, which are really not interesting.
> > Jie