Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - VOTE: HDFS-347 merge


Copy link to this message
-
Re: VOTE: HDFS-347 merge
Suresh Srinivas 2013-02-21, 00:47
On Wed, Feb 20, 2013 at 4:29 PM, Chris Douglas <[EMAIL PROTECTED]> wrote:

> The throughput on this thread is too high.
>
> Nicholas/Suresh: do either of you disagree with the general approach
> in HDFS-347? Holding an inevitable branch open causes a lot of pain
> (Suresh, you endured much of that personally with security, which had
> a lot of follow-on work). While it makes sense to block HDFS-347 from
> 2.x before all concerns are addressed, are most objections so
> fundamental that the merge to trunk should be prevented (except for
> HDFS-2246)? Is there related development that will be impacted? Colin
> won't disappear after this is committed, so if the answers to these
> questions are "no", then let's move forward.
>

Yes. I have already stated this earlier.

"... if the changes are deemed to be trivial, we can do it post merge to
trunk."
> Todd: Yes, the idea that the project only allows optimizations that
> work on all platforms is idiotic, which is why nobody has suggested
> it. Given that HDFS-347 is a strictly better approach, once committed,
> there will be ample motivation to add support for other OSes and
> remove HDFS-2246 entirely. Nobody is confused about this. There's
> ample precedent for retaining obscure, clumsy features as a temporary
> stop-gap (e.g., service plugins, opaque blobs of bytes in Tasks,
> configurable combiner semantics). What's the virtue of insisting on
> removing this? Unless there was a lot of follow-on work, HDFS-2246
> doesn't look like a lot of code... -C
>
> On Wed, Feb 20, 2013 at 3:40 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> > On Wed, Feb 20, 2013 at 3:31 PM, Suresh Srinivas <[EMAIL PROTECTED]>
> wrote:
> >>> Given that this is an optimization, and we have a ton of optimizations
> >>> which don't yet run on Windows, I don't think that should be
> >>> considered. Additionally, the Windows support has not yet been merged,
> >>> nor is it in any release, so this isn't a regression.
> >>>
> >>
> >> This is a critical functionality for HBase peformance and an
> optimization
> >> we consider
> >> very important to have.
> >
> > Too bad it doesn't work in any of the normal installations... none of
> > the packages for Hadoop would allow it to work, given that the data
> > directories will be owned by HDFS and not world readable, and
> > tasks/HBase would run as an "hbase" user, which wouldn't have direct
> > access to the block files.
> >
> >>>
> >>> I would be happy to review an addition to the HDFS-347 branch which
> >>> addresses this issue. But I don't think we should be maintaining two
> >>> codepaths for the sake of an optimization on a platform which is not
> >>> yet officially supported on trunk, especially when the old code path
> >>> is _insecure_ and thus unusable in most environments.
> >>
> >>
> >> I have to disagree. No where in the jira or the design it is explicitly
> >> stated that
> >> the old short circuit functionality is being removed. My assumption has
> been
> >> that it will not be removed.
> >
> > I've tried this avenue in the past on other insecurities which were
> > fixed. Sorry if you were depending on insecure behavior. The project
> > should move on and not have 3+ ways of implementing the same thing.
> >
> >> As regards "officially supported", we have been doing
> >> windows development for
> >> more than a year. In fact branch-1-win is being used by a lot users.
> Given
> >> merging it to branch-1 requires first making it available in trunk, we
> have
> >> been doing
> >> a lot of work in branch-trunk-win. It is almost ready to be merged as
> well.
> >>
> >> I am -1 on removing existing short circuit until an alternative short
> >> circuit similar
> >> to HDFS-347 on all the platforms.
> >
> > Great -- are you committed to building this equivalent feature for
> > Windows, then? On what timeline? From my viewpoint, Windows isn't a
> > supported platform *right now*, so vetoing based on it seems
> > meritless.
> >
> > BTW, the posix_fadvise based readahead is an important optimization

http://hortonworks.com/download/