Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> VOTE: HDFS-347 merge


+
Colin McCabe 2013-02-17, 21:48
+
Andrew Purtell 2013-02-18, 02:35
+
Stack 2013-02-18, 01:49
+
Tsz Wo Sze 2013-02-18, 22:03
+
Todd Lipcon 2013-02-20, 00:11
+
Patrick Angeles 2013-02-20, 18:08
+
Tsz Wo Sze 2013-02-20, 19:56
+
Tsz Wo Sze 2013-02-20, 23:01
+
Todd Lipcon 2013-02-20, 23:06
+
Todd Lipcon 2013-02-20, 23:06
+
Tsz Wo Sze 2013-02-20, 23:08
+
Todd Lipcon 2013-02-20, 23:13
+
Suresh Srinivas 2013-02-20, 23:31
+
Todd Lipcon 2013-02-20, 23:40
+
Suresh Srinivas 2013-02-21, 00:04
+
Todd Lipcon 2013-02-21, 00:12
+
Suresh Srinivas 2013-02-21, 00:28
Copy link to this message
-
Re: VOTE: HDFS-347 merge
The throughput on this thread is too high.

Nicholas/Suresh: do either of you disagree with the general approach
in HDFS-347? Holding an inevitable branch open causes a lot of pain
(Suresh, you endured much of that personally with security, which had
a lot of follow-on work). While it makes sense to block HDFS-347 from
2.x before all concerns are addressed, are most objections so
fundamental that the merge to trunk should be prevented (except for
HDFS-2246)? Is there related development that will be impacted? Colin
won't disappear after this is committed, so if the answers to these
questions are "no", then let's move forward.

Todd: Yes, the idea that the project only allows optimizations that
work on all platforms is idiotic, which is why nobody has suggested
it. Given that HDFS-347 is a strictly better approach, once committed,
there will be ample motivation to add support for other OSes and
remove HDFS-2246 entirely. Nobody is confused about this. There's
ample precedent for retaining obscure, clumsy features as a temporary
stop-gap (e.g., service plugins, opaque blobs of bytes in Tasks,
configurable combiner semantics). What's the virtue of insisting on
removing this? Unless there was a lot of follow-on work, HDFS-2246
doesn't look like a lot of code... -C

On Wed, Feb 20, 2013 at 3:40 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> On Wed, Feb 20, 2013 at 3:31 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
>>> Given that this is an optimization, and we have a ton of optimizations
>>> which don't yet run on Windows, I don't think that should be
>>> considered. Additionally, the Windows support has not yet been merged,
>>> nor is it in any release, so this isn't a regression.
>>>
>>
>> This is a critical functionality for HBase peformance and an optimization
>> we consider
>> very important to have.
>
> Too bad it doesn't work in any of the normal installations... none of
> the packages for Hadoop would allow it to work, given that the data
> directories will be owned by HDFS and not world readable, and
> tasks/HBase would run as an "hbase" user, which wouldn't have direct
> access to the block files.
>
>>>
>>> I would be happy to review an addition to the HDFS-347 branch which
>>> addresses this issue. But I don't think we should be maintaining two
>>> codepaths for the sake of an optimization on a platform which is not
>>> yet officially supported on trunk, especially when the old code path
>>> is _insecure_ and thus unusable in most environments.
>>
>>
>> I have to disagree. No where in the jira or the design it is explicitly
>> stated that
>> the old short circuit functionality is being removed. My assumption has been
>> that it will not be removed.
>
> I've tried this avenue in the past on other insecurities which were
> fixed. Sorry if you were depending on insecure behavior. The project
> should move on and not have 3+ ways of implementing the same thing.
>
>> As regards "officially supported", we have been doing
>> windows development for
>> more than a year. In fact branch-1-win is being used by a lot users. Given
>> merging it to branch-1 requires first making it available in trunk, we have
>> been doing
>> a lot of work in branch-trunk-win. It is almost ready to be merged as well.
>>
>> I am -1 on removing existing short circuit until an alternative short
>> circuit similar
>> to HDFS-347 on all the platforms.
>
> Great -- are you committed to building this equivalent feature for
> Windows, then? On what timeline? From my viewpoint, Windows isn't a
> supported platform *right now*, so vetoing based on it seems
> meritless.
>
> BTW, the posix_fadvise based readahead is an important optimization
> for many workloads, too, but from what I can tell looking at the
> Windows branch, it doesn't support it. There are other places in the
> Windows branch where performance is going to be worse - eg disabling
> the pipelined crc32c implementation will be a 2-3x hit on that code
> path.
>
> No one has voted to merge Windows support, and if merging Windows

On Wed, Feb 20, 2013 at 3:40 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
+
Aaron T. Myers 2013-02-21, 01:12
+
Suresh Srinivas 2013-02-21, 15:40
+
Andrew Purtell 2013-02-21, 01:32
+
Chris Douglas 2013-02-21, 21:24
+
Tsz Wo Sze 2013-02-21, 22:15
+
Eli Collins 2013-02-22, 21:55
+
Tsz Wo Sze 2013-02-23, 02:32
+
Aaron T. Myers 2013-02-23, 02:40
+
Tsz Wo Sze 2013-02-24, 00:23
+
Eli Collins 2013-02-25, 18:24
+
Tsz Wo Sze 2013-02-25, 20:50
+
Eli Collins 2013-02-25, 21:16
+
Chris Douglas 2013-02-25, 21:50
+
Suresh Srinivas 2013-02-26, 00:09
+
Eli Collins 2013-02-26, 00:39
+
Suresh Srinivas 2013-02-26, 17:33
+
Eli Collins 2013-02-26, 19:24
+
Suresh Srinivas 2013-02-26, 19:35
+
Eli Collins 2013-02-26, 21:51
+
Chris Douglas 2013-02-27, 00:52
+
Suresh Srinivas 2013-02-27, 01:09
+
Colin McCabe 2013-03-05, 20:24
+
Suresh Srinivas 2013-03-05, 21:09
+
Tsz Wo Sze 2013-03-05, 23:08
+
sanjay Radia 2013-02-27, 19:45
+
Eli Collins 2013-02-27, 20:06
+
Colin McCabe 2013-02-27, 23:28
+
Eli Collins 2013-02-27, 23:42
+
Chris Douglas 2013-02-27, 23:29
+
Eli Collins 2013-02-25, 22:01
+
Colin McCabe 2013-02-25, 18:31
+
Colin McCabe 2013-02-22, 19:13
+
sanjay Radia 2013-02-27, 01:36
+
Suresh Srinivas 2013-02-21, 00:47
+
Todd Lipcon 2013-02-20, 20:16
+
Suresh Srinivas 2013-02-20, 22:49
+
Todd Lipcon 2013-02-20, 23:01
+
Suresh Srinivas 2013-02-20, 23:19
+
Tsz Wo Sze 2013-02-20, 21:48
+
Todd Lipcon 2013-02-20, 22:27
+
Bikas Saha 2013-02-26, 21:47
+
Todd Lipcon 2013-02-26, 22:07
+
Colin McCabe 2013-04-01, 23:32