Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> VOTE: HDFS-347 merge


+
Colin McCabe 2013-02-17, 21:48
+
Andrew Purtell 2013-02-18, 02:35
+
Stack 2013-02-18, 01:49
+
Tsz Wo Sze 2013-02-18, 22:03
+
Todd Lipcon 2013-02-20, 00:11
+
Patrick Angeles 2013-02-20, 18:08
+
Tsz Wo Sze 2013-02-20, 19:56
+
Tsz Wo Sze 2013-02-20, 23:01
+
Todd Lipcon 2013-02-20, 23:06
+
Todd Lipcon 2013-02-20, 23:06
+
Tsz Wo Sze 2013-02-20, 23:08
+
Todd Lipcon 2013-02-20, 23:13
+
Suresh Srinivas 2013-02-20, 23:31
+
Todd Lipcon 2013-02-20, 23:40
+
Suresh Srinivas 2013-02-21, 00:04
+
Todd Lipcon 2013-02-21, 00:12
+
Suresh Srinivas 2013-02-21, 00:28
+
Chris Douglas 2013-02-21, 00:29
+
Aaron T. Myers 2013-02-21, 01:12
+
Suresh Srinivas 2013-02-21, 15:40
+
Andrew Purtell 2013-02-21, 01:32
+
Chris Douglas 2013-02-21, 21:24
+
Tsz Wo Sze 2013-02-21, 22:15
+
Eli Collins 2013-02-22, 21:55
+
Tsz Wo Sze 2013-02-23, 02:32
+
Aaron T. Myers 2013-02-23, 02:40
+
Tsz Wo Sze 2013-02-24, 00:23
+
Eli Collins 2013-02-25, 18:24
+
Tsz Wo Sze 2013-02-25, 20:50
+
Eli Collins 2013-02-25, 21:16
+
Chris Douglas 2013-02-25, 21:50
+
Suresh Srinivas 2013-02-26, 00:09
+
Eli Collins 2013-02-26, 00:39
+
Suresh Srinivas 2013-02-26, 17:33
+
Eli Collins 2013-02-26, 19:24
+
Suresh Srinivas 2013-02-26, 19:35
+
Eli Collins 2013-02-26, 21:51
+
Chris Douglas 2013-02-27, 00:52
+
Suresh Srinivas 2013-02-27, 01:09
+
Colin McCabe 2013-03-05, 20:24
+
Suresh Srinivas 2013-03-05, 21:09
+
Tsz Wo Sze 2013-03-05, 23:08
+
sanjay Radia 2013-02-27, 19:45
+
Eli Collins 2013-02-27, 20:06
+
Colin McCabe 2013-02-27, 23:28
+
Eli Collins 2013-02-27, 23:42
+
Chris Douglas 2013-02-27, 23:29
+
Eli Collins 2013-02-25, 22:01
+
Colin McCabe 2013-02-25, 18:31
+
Colin McCabe 2013-02-22, 19:13
+
sanjay Radia 2013-02-27, 01:36
Copy link to this message
-
Re: VOTE: HDFS-347 merge
On Wed, Feb 20, 2013 at 4:29 PM, Chris Douglas <[EMAIL PROTECTED]> wrote:

> The throughput on this thread is too high.
>
> Nicholas/Suresh: do either of you disagree with the general approach
> in HDFS-347? Holding an inevitable branch open causes a lot of pain
> (Suresh, you endured much of that personally with security, which had
> a lot of follow-on work). While it makes sense to block HDFS-347 from
> 2.x before all concerns are addressed, are most objections so
> fundamental that the merge to trunk should be prevented (except for
> HDFS-2246)? Is there related development that will be impacted? Colin
> won't disappear after this is committed, so if the answers to these
> questions are "no", then let's move forward.
>

Yes. I have already stated this earlier.

"... if the changes are deemed to be trivial, we can do it post merge to
trunk."
> Todd: Yes, the idea that the project only allows optimizations that
> work on all platforms is idiotic, which is why nobody has suggested
> it. Given that HDFS-347 is a strictly better approach, once committed,
> there will be ample motivation to add support for other OSes and
> remove HDFS-2246 entirely. Nobody is confused about this. There's
> ample precedent for retaining obscure, clumsy features as a temporary
> stop-gap (e.g., service plugins, opaque blobs of bytes in Tasks,
> configurable combiner semantics). What's the virtue of insisting on
> removing this? Unless there was a lot of follow-on work, HDFS-2246
> doesn't look like a lot of code... -C
>
> On Wed, Feb 20, 2013 at 3:40 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> > On Wed, Feb 20, 2013 at 3:31 PM, Suresh Srinivas <[EMAIL PROTECTED]>
> wrote:
> >>> Given that this is an optimization, and we have a ton of optimizations
> >>> which don't yet run on Windows, I don't think that should be
> >>> considered. Additionally, the Windows support has not yet been merged,
> >>> nor is it in any release, so this isn't a regression.
> >>>
> >>
> >> This is a critical functionality for HBase peformance and an
> optimization
> >> we consider
> >> very important to have.
> >
> > Too bad it doesn't work in any of the normal installations... none of
> > the packages for Hadoop would allow it to work, given that the data
> > directories will be owned by HDFS and not world readable, and
> > tasks/HBase would run as an "hbase" user, which wouldn't have direct
> > access to the block files.
> >
> >>>
> >>> I would be happy to review an addition to the HDFS-347 branch which
> >>> addresses this issue. But I don't think we should be maintaining two
> >>> codepaths for the sake of an optimization on a platform which is not
> >>> yet officially supported on trunk, especially when the old code path
> >>> is _insecure_ and thus unusable in most environments.
> >>
> >>
> >> I have to disagree. No where in the jira or the design it is explicitly
> >> stated that
> >> the old short circuit functionality is being removed. My assumption has
> been
> >> that it will not be removed.
> >
> > I've tried this avenue in the past on other insecurities which were
> > fixed. Sorry if you were depending on insecure behavior. The project
> > should move on and not have 3+ ways of implementing the same thing.
> >
> >> As regards "officially supported", we have been doing
> >> windows development for
> >> more than a year. In fact branch-1-win is being used by a lot users.
> Given
> >> merging it to branch-1 requires first making it available in trunk, we
> have
> >> been doing
> >> a lot of work in branch-trunk-win. It is almost ready to be merged as
> well.
> >>
> >> I am -1 on removing existing short circuit until an alternative short
> >> circuit similar
> >> to HDFS-347 on all the platforms.
> >
> > Great -- are you committed to building this equivalent feature for
> > Windows, then? On what timeline? From my viewpoint, Windows isn't a
> > supported platform *right now*, so vetoing based on it seems
> > meritless.
> >
> > BTW, the posix_fadvise based readahead is an important optimization

http://hortonworks.com/download/
+
Todd Lipcon 2013-02-20, 20:16
+
Suresh Srinivas 2013-02-20, 22:49
+
Todd Lipcon 2013-02-20, 23:01
+
Suresh Srinivas 2013-02-20, 23:19
+
Tsz Wo Sze 2013-02-20, 21:48
+
Todd Lipcon 2013-02-20, 22:27
+
Bikas Saha 2013-02-26, 21:47
+
Todd Lipcon 2013-02-26, 22:07
+
Colin McCabe 2013-04-01, 23:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB