Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> VOTE: HDFS-347 merge

Copy link to this message
Re: VOTE: HDFS-347 merge
> >
> > I have to disagree. No where in the jira or the design it is explicitly
> > stated that
> > the old short circuit functionality is being removed. My assumption has
> been
> > that it will not be removed.
> I've tried this avenue in the past on other insecurities which were
> fixed. Sorry if you were depending on insecure behavior. The project
> should move on and not have 3+ ways of implementing the same thing.

Todd, can you please show in the jira or design document where
explicitly it is stated that old short circuit will be removed.

I agree that we need to move past the old short circuit, but not now.

Great -- are you committed to building this equivalent feature for
> Windows, then? On what timeline?
Yes. I plan to get the equivalent functionality in a couple of months, once
the existing work on snapshots and some HA related cleanup, RPC cleanup
completes. I plan to make it available in a dot release after 2.x goes GA.
> From my viewpoint, Windows isn't a
> supported platform *right now*, so vetoing based on it seems
> meritless.

I disagree. If I had timed branch-trunk-win merge before HDFS-347, then
would you consider windows a supported platform? You know
that more than a years worth of work has gone into windows support,
all in the public.
> BTW, the posix_fadvise based readahead is an important optimization
> for many workloads, too, but from what I can tell looking at the
> Windows branch, it doesn't support it. There are other places in the
> Windows branch where performance is going to be worse - eg disabling
> the pipelined crc32c implementation will be a 2-3x hit on that code
> path.

We will add similar optimizations as well. You are not seeing the subtle
difference. You are removing a functionality that is already working on
windows. These optimizations are not available on windows currently,
given in Hadoop we have been making optimizations for *nix for a long time.
They will become available in due course of time. We could certainly
discuss removing this functionality then. I also offer to support two code
paths until then.
> No one has voted to merge Windows support, and if merging Windows
> support means that, from now on, every _optimization_ must work on
> Windows, I don't think I will be able to vote +1. The vast majority of
> the community is _not_ running Windows, and I don't want to block
> progress on the small number of developers who know how to program
> against that platform.

I have only sent heads up email about coming merge. It is not an email
asking for vote. The vast majority of the community is not running
windows because it was not supported well. That is changing with the
work that is going on in windows branch.

Again you seem to be misunderstanding my comment.  I am not asking
for *every_optimization_must work on windows*. You are free to optimize
for a platform as you want. All I am asking for not removing an optimization
that is already available on windows.
> If that's the axe hanging over our head with the Windows branch, then
> I'm all for saying "good, keep it on a branch and don't merge it to
> trunk".
> I was hoping we could all work together a bit better here...
> contentious merge votes like this just cause cases where different
> distros diverge by merging different branches way ahead of the
> upstream (eg yours with windows support, ours with 347, etc)
HDFS-347 does not clearly state old short circuit will be removed any where
in the jira or design. If this was made clear in the jira, this discussion
have happened much earlier than now.

You seem to be taking the comments I am making the wrong way. I am
supportive of this work. In fact as you see some of us have spent time
testing this work and have reviewed the code.