Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Some notes from the HBase developer Pow-Wow

Copy link to this message
Some notes from the HBase developer Pow-Wow
- Stabilizing 0.96 / CI
  -- not running continuously
  -- Roman:   is it a good idea to run IT under Bigtop
       - the HBase unit tests are good..
       - What about HBase as a backend for hive - tests for those
       - Do we think there is value with the integration with the rest
of the hadoop ecosystem
  -- Jon: What's needed to make this work
  -- Roman: Collective agreement that we want to solve this
  -- Stack: We need to run the IT tests from Enis and Sergey running
  -- Roman: If we all agree that this is something needs to be fixed
.. then yes we would talk about mechanics. Bigtop doesn't have
expertise in HBase and hence HBase folks would have to debug failures.
  -- Roman: Bigtop is committed to tests. Less than a dozen tests for
hbase currently..
  -- We have been running validation around RC time. Find all kinds of
issues - sometimes trivial (maybe a config issue),
  -- Roman can offer CI for trunk but will work for only hadoop-2 line.
  -- Roman does first line of triage.
  -- No issues to do with other ecosystem artifacts. Bigtop ensures
the right artifacts are in place.
  -- hadoop-2 is important but not particular about the version of
hadoop within 2. For example 2.0.2
  -- Gary: Will be good to run security tests
  -- Roman & Devaraj to talk on how this can be done/implemented

On 0.96 branching
- Lars:
   -- We will have three branches to maintain.
- Stack: we need to stabilize quickly
- Enis: What about 0.95.
- Stack: Just do the snapshots thing. Every week, give a snapshot
- Enis: we've done a bunch of stuff in RPC.. If we have to break
something, we can if it is beta (0.96-beta).
- Agreement is there generally ... Debate on the name with snapshot
versus 95.0/96.0
  -- 0.95 experimental .. 0.96 will be stable
  -- If we go with 0.95, releasing will be easier as well..
  -- 0.95 will not be in production..
  -- 0.96 will be off 0.95 branch and not trunk based

- How do we go about committing issues.. (issue commit rate is low)
 -- 0.94 is stable -  2 new commits and 2 new bugs a day
 -- 0.96 has lots of issues not reviewed
 -- Break up the patch into multiple smaller pieces to make review easier
 -- Branching on a big feature was suggested -
   --- Issues: committer needs to be there
         Sergey: If the rate of change is high on common code (to the
branches) then merging will be tough
   --- Jon: Refactor should be done in the main branch (since it
doesn't add any new funtionality)
   --- Release often to reduce #backports overall and issues with that..

- Review process .. how to drive a review to closure. Effort goes
waste if the review process is not completed. The same reviewer should
continue to review the patch ..
- Hard to enforce any process
- Enis: there should be a summary of the patch and all that .. so that
the review process is helped.. Hard to understand the architecture of
the patch unless documented
- Jon: It should be easy to make a one-to-one correspondence between
the description and the patch
- The commit should have only the jira# as opposed to pages of description

- Component owners: is this working. Committers need to be forthcoming
with reviews
  -- Maybe review the modules and add some more if needed.

  -- Good that we have more contributions coming than we have
reviewers, but unless we keep up, we will plateau
  -- Mail on dev@ list if review doesn't happen

- Dev co-ordination:
 -- How best can we pull together
 -- Priorities:
   --- Getting 0.96 out is priority
   --- Backports to 0.94 will happen .. until 0.96 is stable
   --- 0.92 release ? Any committer who wants to make a release can do
so (maybe with some backports, etc.)
   --- Backporting can be tough if there are bugs and the bugfix has
to be applied to all branches

- HBASE-2600 - this requires a change in the client and the server.
They have to be changed in lock-step. Its hard to do this .. Jon
doesn't want to have the fix for 0.96. So 0.98 might be another
singular release. Maybe do a rewrite of the meta after taking a lock
on the meta, do a shim layer to handle the backward compat. between
0.96 and 0.98

- What do people want to get into 0.94
 -- The biggest thing - Snapshots, mostly new code, about a 3rd of the
stuff in 0.94 already
 -- Compactions improvments - no backport

- How devs can better co-ordinate
  -- Snapshots co-ordination working well
  -- One page design is useful (makes it readable and all)
  -- How about handling the stripe compaction - where an idea leads to
a bunch of others
  -- Again write-up should be done

- Should we change the description to match the comments
  -- Two ideas suggested:
   ---  We probably should have the description updated with the
"Date: new description" if the issue at hand is updated
   --- Should we have a summary after a bunch of comments - yes

- The face-to-face meetings are useful. We should semd out the minutes
of the discussion to the dev list. We probably should have more
focused huddles. Discuss but don't decide (decide on the jira)!

- Jon: Would people be amenable to merge sooner rather than later on
snapshots? Tested and being beaten up.
- Stack: Yes

- What else goes in in 0.96:
  -- RPC refactor
  -- ROOT removal
  -- Compaction stuff
  -- Package name mailing list thread - there is now a jira on that.
We shouldn't break clients. Package name changes is not worth the

- A bunch of discussions on the RPC with KeyValue/Cells

- What do we do about usability
  -- It'll be nice if we don't need to change configs..
  -- Maybe expose more metrics and then allow for online config
changes since automatic config is difficult and needs to be battle
tested and all.

- Benchmarking of the release:
  -- We should measure the overhead of PB stuff
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB