Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> VOTE: HDFS-347 merge


+
Colin McCabe 2013-02-17, 21:48
+
Andrew Purtell 2013-02-18, 02:35
+
Stack 2013-02-18, 01:49
+
Tsz Wo Sze 2013-02-18, 22:03
+
Todd Lipcon 2013-02-20, 00:11
+
Patrick Angeles 2013-02-20, 18:08
+
Tsz Wo Sze 2013-02-20, 19:56
+
Tsz Wo Sze 2013-02-20, 23:01
+
Todd Lipcon 2013-02-20, 23:06
+
Todd Lipcon 2013-02-20, 23:06
+
Tsz Wo Sze 2013-02-20, 23:08
+
Todd Lipcon 2013-02-20, 23:13
+
Suresh Srinivas 2013-02-20, 23:31
+
Todd Lipcon 2013-02-20, 23:40
+
Suresh Srinivas 2013-02-21, 00:04
+
Todd Lipcon 2013-02-21, 00:12
+
Suresh Srinivas 2013-02-21, 00:28
+
Chris Douglas 2013-02-21, 00:29
+
Aaron T. Myers 2013-02-21, 01:12
+
Suresh Srinivas 2013-02-21, 15:40
+
Andrew Purtell 2013-02-21, 01:32
+
Chris Douglas 2013-02-21, 21:24
+
Tsz Wo Sze 2013-02-21, 22:15
+
Eli Collins 2013-02-22, 21:55
+
Tsz Wo Sze 2013-02-23, 02:32
+
Aaron T. Myers 2013-02-23, 02:40
+
Tsz Wo Sze 2013-02-24, 00:23
+
Eli Collins 2013-02-25, 18:24
+
Tsz Wo Sze 2013-02-25, 20:50
+
Eli Collins 2013-02-25, 21:16
+
Chris Douglas 2013-02-25, 21:50
+
Suresh Srinivas 2013-02-26, 00:09
+
Eli Collins 2013-02-26, 00:39
+
Suresh Srinivas 2013-02-26, 17:33
+
Eli Collins 2013-02-26, 19:24
+
Suresh Srinivas 2013-02-26, 19:35
+
Eli Collins 2013-02-26, 21:51
+
Chris Douglas 2013-02-27, 00:52
+
Suresh Srinivas 2013-02-27, 01:09
+
Colin McCabe 2013-03-05, 20:24
+
Suresh Srinivas 2013-03-05, 21:09
+
Tsz Wo Sze 2013-03-05, 23:08
+
sanjay Radia 2013-02-27, 19:45
+
Eli Collins 2013-02-27, 20:06
+
Colin McCabe 2013-02-27, 23:28
+
Eli Collins 2013-02-27, 23:42
+
Chris Douglas 2013-02-27, 23:29
+
Eli Collins 2013-02-25, 22:01
+
Colin McCabe 2013-02-25, 18:31
+
Colin McCabe 2013-02-22, 19:13
+
sanjay Radia 2013-02-27, 01:36
Copy link to this message
-
Re: VOTE: HDFS-347 merge
On Wed, Feb 20, 2013 at 4:29 PM, Chris Douglas <[EMAIL PROTECTED]> wrote:

> The throughput on this thread is too high.
>
> Nicholas/Suresh: do either of you disagree with the general approach
> in HDFS-347? Holding an inevitable branch open causes a lot of pain
> (Suresh, you endured much of that personally with security, which had
> a lot of follow-on work). While it makes sense to block HDFS-347 from
> 2.x before all concerns are addressed, are most objections so
> fundamental that the merge to trunk should be prevented (except for
> HDFS-2246)? Is there related development that will be impacted? Colin
> won't disappear after this is committed, so if the answers to these
> questions are "no", then let's move forward.
>

Yes. I have already stated this earlier.

"... if the changes are deemed to be trivial, we can do it post merge to
trunk."
> Todd: Yes, the idea that the project only allows optimizations that
> work on all platforms is idiotic, which is why nobody has suggested
> it. Given that HDFS-347 is a strictly better approach, once committed,
> there will be ample motivation to add support for other OSes and
> remove HDFS-2246 entirely. Nobody is confused about this. There's
> ample precedent for retaining obscure, clumsy features as a temporary
> stop-gap (e.g., service plugins, opaque blobs of bytes in Tasks,
> configurable combiner semantics). What's the virtue of insisting on
> removing this? Unless there was a lot of follow-on work, HDFS-2246
> doesn't look like a lot of code... -C
>
> On Wed, Feb 20, 2013 at 3:40 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> > On Wed, Feb 20, 2013 at 3:31 PM, Suresh Srinivas <[EMAIL PROTECTED]>
> wrote:
> >>> Given that this is an optimization, and we have a ton of optimizations
> >>> which don't yet run on Windows, I don't think that should be
> >>> considered. Additionally, the Windows support has not yet been merged,
> >>> nor is it in any release, so this isn't a regression.
> >>>
> >>
> >> This is a critical functionality for HBase peformance and an
> optimization
> >> we consider
> >> very important to have.
> >
> > Too bad it doesn't work in any of the normal installations... none of
> > the packages for Hadoop would allow it to work, given that the data
> > directories will be owned by HDFS and not world readable, and
> > tasks/HBase would run as an "hbase" user, which wouldn't have direct
> > access to the block files.
> >
> >>>
> >>> I would be happy to review an addition to the HDFS-347 branch which
> >>> addresses this issue. But I don't think we should be maintaining two
> >>> codepaths for the sake of an optimization on a platform which is not
> >>> yet officially supported on trunk, especially when the old code path
> >>> is _insecure_ and thus unusable in most environments.
> >>
> >>
> >> I have to disagree. No where in the jira or the design it is explicitly
> >> stated that
> >> the old short circuit functionality is being removed. My assumption has
> been
> >> that it will not be removed.
> >
> > I've tried this avenue in the past on other insecurities which were
> > fixed. Sorry if you were depending on insecure behavior. The project
> > should move on and not have 3+ ways of implementing the same thing.
> >
> >> As regards "officially supported", we have been doing
> >> windows development for
> >> more than a year. In fact branch-1-win is being used by a lot users.
> Given
> >> merging it to branch-1 requires first making it available in trunk, we
> have
> >> been doing
> >> a lot of work in branch-trunk-win. It is almost ready to be merged as
> well.
> >>
> >> I am -1 on removing existing short circuit until an alternative short
> >> circuit similar
> >> to HDFS-347 on all the platforms.
> >
> > Great -- are you committed to building this equivalent feature for
> > Windows, then? On what timeline? From my viewpoint, Windows isn't a
> > supported platform *right now*, so vetoing based on it seems
> > meritless.
> >
> > BTW, the posix_fadvise based readahead is an important optimization

http://hortonworks.com/download/
+
Todd Lipcon 2013-02-20, 20:16
+
Suresh Srinivas 2013-02-20, 22:49
+
Todd Lipcon 2013-02-20, 23:01
+
Suresh Srinivas 2013-02-20, 23:19
+
Tsz Wo Sze 2013-02-20, 21:48
+
Todd Lipcon 2013-02-20, 22:27
+
Bikas Saha 2013-02-26, 21:47
+
Todd Lipcon 2013-02-26, 22:07
+
Colin McCabe 2013-04-01, 23:32