Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk


Copy link to this message
-
Re: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk
+1 nice feature for HDFS
On Fri, Dec 6, 2013 at 7:32 PM, Arpit Agarwal <[EMAIL PROTECTED]>wrote:

> Hi Andrew,
>
> Our plan as stated back in August was to do this work principally in two
> phases.
> https://issues.apache.org/jira/browse/HDFS-2832?focusedCommentId=13739041
>
> For the second phase which includes API support, we also need quota
> management. For changes of this scope, to do all the work at once while
> keeping the feature branch in sync with ongoing development in trunk is
> unmanageable. Hence we'd like to stick with the initial plan and develop in
> phases.
>
> Even for datanode caching the initial merge did not include the quota
> management changes which are happening subsequently.
>
> Going forward, we will stabilize the current changes in trunk in the 2.4
> time frame. Next we will add quota management and API support which can
> align with the 2.5 time frame, with the second merge potentially in
> March/April.
>
> Arpit
>
>
> On Fri, Dec 6, 2013 at 3:15 PM, Andrew Wang <[EMAIL PROTECTED]
> >wrote:
>
> > Hi everyone,
> >
> > I'm still getting up to speed on the changes here (my fault for not
> > following development more closely, other priorities etc etc), but the
> > branch thus far is already quite impressive. It's quite an undertaking to
> > turn the DN into a collection of Storages, along with the corresponding
> > datastructure, tracking, and other changes in the NN and DN.
> >
> > Correct me if I'm wrong though, but this still leaves a substantial part
> of
> > the design doc to be implemented. Looking at the list of remaining
> > subtasks, it seems like we still can't specify a storage type for a file
> > (HDFS-5229) or write a file to a given storage type (HDFS-5391), along
> with
> > the corresponding client protocol changes. This leads me to two
> questions:
> >
> > - If this is merged, what can I do with the new code? Without client
> > changes or the ability to create a file on a different storage type, I
> > don't know how (for example) I could hand this to our QA team to test.
> I'm
> > wondering why we want to merge now rather than when the branch is more
> > feature complete.
> > - What's the plan for the implementation of the remaining features? How
> > many phases? What's the timeline for these phases? Particularly, related
> to
> > the use cases presented in section 2 of the design doc.
> >
> > I'm also going to post some design doc questions to the JIRA, there are a
> > few technical q's I'd like to get clarification on.
> >
> > Thanks,
> > Andrew
> >
> >
> > On Wed, Dec 4, 2013 at 7:21 AM, Sirianni, Eric <[EMAIL PROTECTED]
> > >wrote:
> >
> > > +1
> > >
> > > My team has been developing and testing against the HDFS-2832 branch
> for
> > > the past month.  It has proven to be quite stable.
> > >
> > > Eric
> > >
> > > -----Original Message-----
> > > From: Arpit Agarwal [mailto:[EMAIL PROTECTED]]
> > > Sent: Monday, December 02, 2013 7:07 PM
> > > To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> > > Subject: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk
> > >
> > > Hello all,
> > >
> > > I would like to call a vote to merge phase 1 of the Heterogeneous
> Storage
> > > feature into trunk.
> > >
> > > *Scope of the changes:*
> > > The changes allow exposing the DataNode as a collection of storages and
> > set
> > > the foundation for subsequent work to present Heterogeneous Storages to
> > > applications. This allows DataNodes to send block and storage reports
> > > per-storage. In addition this change introduces the ability to add a
> > > 'storage type' tag to the storage directories. This enables supporting
> > > different types of storages in addition to disk storage.
> > >
> > > Development of the feature is tracked in the jira
> > > https://issues.apache.org/jira/browse/HDFS-2832.
> > >
> > > *Details of development and testing:*
> > > Development has been done in a separate branch -
> > > https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2832. The
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB