Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk


Copy link to this message
-
Re: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk
+1 nice feature for HDFS
On Fri, Dec 6, 2013 at 7:32 PM, Arpit Agarwal <[EMAIL PROTECTED]>wrote:

> Hi Andrew,
>
> Our plan as stated back in August was to do this work principally in two
> phases.
> https://issues.apache.org/jira/browse/HDFS-2832?focusedCommentId=13739041
>
> For the second phase which includes API support, we also need quota
> management. For changes of this scope, to do all the work at once while
> keeping the feature branch in sync with ongoing development in trunk is
> unmanageable. Hence we'd like to stick with the initial plan and develop in
> phases.
>
> Even for datanode caching the initial merge did not include the quota
> management changes which are happening subsequently.
>
> Going forward, we will stabilize the current changes in trunk in the 2.4
> time frame. Next we will add quota management and API support which can
> align with the 2.5 time frame, with the second merge potentially in
> March/April.
>
> Arpit
>
>
> On Fri, Dec 6, 2013 at 3:15 PM, Andrew Wang <[EMAIL PROTECTED]
> >wrote:
>
> > Hi everyone,
> >
> > I'm still getting up to speed on the changes here (my fault for not
> > following development more closely, other priorities etc etc), but the
> > branch thus far is already quite impressive. It's quite an undertaking to
> > turn the DN into a collection of Storages, along with the corresponding
> > datastructure, tracking, and other changes in the NN and DN.
> >
> > Correct me if I'm wrong though, but this still leaves a substantial part
> of
> > the design doc to be implemented. Looking at the list of remaining
> > subtasks, it seems like we still can't specify a storage type for a file
> > (HDFS-5229) or write a file to a given storage type (HDFS-5391), along
> with
> > the corresponding client protocol changes. This leads me to two
> questions:
> >
> > - If this is merged, what can I do with the new code? Without client
> > changes or the ability to create a file on a different storage type, I
> > don't know how (for example) I could hand this to our QA team to test.
> I'm
> > wondering why we want to merge now rather than when the branch is more
> > feature complete.
> > - What's the plan for the implementation of the remaining features? How
> > many phases? What's the timeline for these phases? Particularly, related
> to
> > the use cases presented in section 2 of the design doc.
> >
> > I'm also going to post some design doc questions to the JIRA, there are a
> > few technical q's I'd like to get clarification on.
> >
> > Thanks,
> > Andrew
> >
> >
> > On Wed, Dec 4, 2013 at 7:21 AM, Sirianni, Eric <[EMAIL PROTECTED]
> > >wrote:
> >
> > > +1
> > >
> > > My team has been developing and testing against the HDFS-2832 branch
> for
> > > the past month.  It has proven to be quite stable.
> > >
> > > Eric
> > >
> > > -----Original Message-----
> > > From: Arpit Agarwal [mailto:[EMAIL PROTECTED]]
> > > Sent: Monday, December 02, 2013 7:07 PM
> > > To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> > > Subject: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk
> > >
> > > Hello all,
> > >
> > > I would like to call a vote to merge phase 1 of the Heterogeneous
> Storage
> > > feature into trunk.
> > >
> > > *Scope of the changes:*
> > > The changes allow exposing the DataNode as a collection of storages and
> > set
> > > the foundation for subsequent work to present Heterogeneous Storages to
> > > applications. This allows DataNodes to send block and storage reports
> > > per-storage. In addition this change introduces the ability to add a
> > > 'storage type' tag to the storage directories. This enables supporting
> > > different types of storages in addition to disk storage.
> > >
> > > Development of the feature is tracked in the jira
> > > https://issues.apache.org/jira/browse/HDFS-2832.
> > >
> > > *Details of development and testing:*
> > > Development has been done in a separate branch -
> > > https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2832. The