Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> [DISCUSS] Remove append?


+
Eli Collins 2012-03-21, 00:37
+
Dave Shine 2012-03-21, 12:36
+
Konstantin Shvachko 2012-03-22, 08:11
+
Tim Broberg 2012-03-21, 18:31
+
Eli Collins 2012-03-21, 18:52
+
Dave Shine 2012-03-21, 19:07
+
Milind.Bhandarkar@... 2012-03-21, 17:17
+
Eli Collins 2012-03-21, 17:32
+
Eli Collins 2012-03-21, 17:33
+
Milind.Bhandarkar@... 2012-03-21, 17:47
+
Eli Collins 2012-03-21, 18:27
+
Milind.Bhandarkar@... 2012-03-21, 19:30
+
Eli Collins 2012-03-21, 21:14
+
Milind.Bhandarkar@... 2012-03-21, 19:48
+
Eli Collins 2012-03-21, 21:09
+
Milind.Bhandarkar@... 2012-03-21, 22:06
+
Eli Collins 2012-03-21, 22:16
+
Milind.Bhandarkar@... 2012-03-21, 22:48
+
Eli Collins 2012-03-21, 23:30
+
Milind.Bhandarkar@... 2012-03-21, 20:24
+
Konstantin Shvachko 2012-03-22, 08:26
+
Eli Collins 2012-03-22, 17:25
+
Scott Carey 2012-03-24, 02:44
+
Colin McCabe 2012-03-26, 19:53
+
Scott Carey 2012-03-26, 20:53
+
Sanjay Radia 2012-03-21, 20:57
+
Eli Collins 2012-03-21, 21:08
+
Tsz Wo Sze 2012-03-21, 20:31
+
Eli Collins 2012-03-21, 20:58
Copy link to this message
-
Re: [DISCUSS] Remove append?
On Mar 20, 2012, at 7:37 PM, Eli Collins wrote:
> Hey gang,
>
> I'd like to get people's thoughts on the following proposal. I think
> we should consider removing append from HDFS.
>
> Where we are today.. append was added in the 0.17-19 releases
> (HADOOP-1700) and subsequently disabled (HADOOP-5224) due to quality
> issues. It and sync were re-designed, re-implemented, and shipped in
> 21.0 (HDFS-265). To my knowledge, there has been no real production
> use. Anecdotally people who worked on branch-20-append have told me
> they think the new trunk code is substantially less well-tested than
> the branch-20-append code (at least for sync, append was never well
> tested). It has certainly gotten way less pounding from HBase users.
> The design however, is much improved, and people think we can get
> hsync (and append) stabilized in trunk (mostly testing and bug
> fixing).

Up front:  I think append is a needed feature.

Politely speaking, I think the premise of the question is a bit dubious due to circular nature.  Ie. It's not used in production so is it worth it?  The stigma/perception that append has been unstable and is not well-tested is a compelling reason to not be in production at major installations.  The situation is going to be akin to "You go first. No, you go first!  No way, you go first!".

Downstream projects also aren't going to use something until it's stable, so they either work around the limitation, or...  they chose something other hdfs.  There's also the unanswerable question of how potential users have been silently lost.  We are unlikely to have heard the user demand from those that chose another solution.  Generally for every complaint/request, a large N-many people didn't even bother.

I envision a day where hdfs is a performant posix filesystem.  Dropping append sets us back from that goal.  Admittedly, I don't know all the intricacies of how append was implemented and why it is/was difficult.  Is the complexity maybe due to "bolting" append onto code that wasn't designed with mutability in mind?  (That's truly a question, not a statement) If so, perhaps a refactoring would simplify the code?

Dropping append also might be used as a cudgel against hdfs.  Cynically speaking, do we want to risk marketeers from certain competitors to say or imply:  Trust your data with us because we're so brilliant that we have a feature hdfs has repeatedly tried and failed to implement!

Daryn
+
Eli Collins 2012-03-22, 17:47
+
Scott Carey 2012-03-24, 02:26
+
Milind.Bhandarkar@... 2012-03-22, 23:27
+
Eli Collins 2012-03-23, 00:41
+
Scott Carey 2012-03-24, 02:46
+
Tsz Wo Sze 2012-03-23, 00:03
+
Eli Collins 2012-03-23, 00:49
+
Colin McCabe 2012-03-26, 20:02
+
Tsz Wo Sze 2012-03-26, 20:55
+
Colin McCabe 2012-03-26, 21:31
+
Tsz Wo Sze 2012-03-27, 02:46
+
Dhruba Borthakur 2012-03-23, 01:18
+
Daryn Sharp 2012-03-23, 17:03
+
CHANG Lei 2012-03-23, 06:22