Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> Update on hadoop-0.23


Copy link to this message
-
Update on hadoop-0.23
Greetings,

I thought I'd drop a note to update folks on progress of hadoop-0.23.

Things are have been very busy in hadoop-0.23 land. We continue to crank through the issues and get ready to ship.

We are mostly pass the initial teething pains of moving our entire build infrastructure to Maven - many thanks to Alejandro, Tom, Giri & Eric Yang.

HDFS is nearly there:
# HDFS Federation and Client side mount tables have been tested with ~300 node clusters with security turned on.
# HDFS upgrades have been tested from 0.20.2xx.
# Functional tests for HDFS  are complete.

NextGen MapReduce (aka MRv2, aka YARN) is coming along great:
# We are happy to report we've done extensive scale testing to confirm stability
  - Sort/GridMixv3 etc. at ~350nodes
  - Scale testing with simulated clusters of ~1500 nodes
# Functional tests for all of MapReduce functionality
# Pig  (0.9 & 0.9.1) working with NextGen MapReduce
# All above have been done with no regressions in security.

We are about to finish performance certification for both HDFS & MapReduce in the next couple of weeks too, after which we start integration tests with HBase, Hive, Oozie etc.

We have cranked through 75 bugs in September alone (http://s.apache.org/mr-sept) and have another 50-ish bugs to go... we have at least 4 different organizations contributing patches to MRv2 in Sept alone: Yahoo, Hortonworks, LinkedIn & Huawei.

Given where we are I'm confident we can have a strong hadoop-0.23.0 release by late October. The current plan is to deploy to alpha clusters in November. Citius, Altius, Fortius! :)

Thanks to everyone who contributed, look forward to continued help.

Arun

PS: I'll continue to provide a periodic updates as we get closer to a hadoop-0.23.0 release.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB