On Wed, Sep 13, 2017 at 1:44 PM Robert Bradshaw <[EMAIL PROTECTED]lid>
I'm not sure, but I don't see why the proposed approach of view.get()
wouldn't work well, or be harder to implement in Python.
I think Kenn is very strongly against using it for state, whereas I don't
have an opinion either way because I can't think of a use case for
accessing state from a lambda - we should probably discuss this separately,
with proposed code examples in front of us.
For side outputs, yes, it might be nice to ".add()" to a PCollection, but
it would require bigger changes - e.g. creating intermediate PCollection's
and inserting an implicit Flatten in front of all steps that contribute to
this PCollection, because a PCollection currently can be produced only by 1
step. Maybe there's a different way to express implicit side outputs.
Either way I support the idea of looking for such a way because it would
simplify use cases such as error handling dead-letter collections.
I guess the bigger point is: do we want to block the discussion of implicit
side inputs on making a decision about the implicitness of other things
(side outputs, state, PipelineOptions, window etc). I can see the argument
for a "yes, block", but can also see the argument for a "no, don't block" -
because this proposal is (as indicated earlier in the thread)
forward-compatible with annotation-based wiring, because we already have a
precedent for implicit access of something via ValueProvider, and because
of the advantages it offers.
Want to mention another advantage: lambdas are likely to be much easier
than NewDoFn approach to use from non-Java but JVM languages/SDKs (e.g.
Scio), which might have even more type erasure, or might have less
sophisticated annotation machinery, or NewDoFn-style anonymous classes
might be highly non-idiomatic in them. Lambdas are idiomatic in every
language that supports lambdas, which these days is basically every
language. [I might be opening a can of worms here, but I guess you can
consider this an argument against NewDoFn in general - though that's
certainly outside the scope of this thread].