Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # dev >> Hadoop 2.0 Support for Accumulo 1.4 Branch


Copy link to this message
-
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
On Fri, Jul 26, 2013 at 2:33 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:

> > If these patches are going to be included with 1.4.4 or 1.4.5, I would
> like
> > to see the following test run using CDH4 on at least a 5 node cluster.
> >  More nodes would be better.
> >
> >   * unit test
> >   * Functional test
> >   * 24 hr Continuous ingest + verification
> >   * 24 hr Continuous ingest + verification + agitation
> >   * 24 hr Random walk
> >   * 24 hr Random walk + agitation
> >
> > I may be able to assist with this, but I can not make any promises.
>
> Sure thing. Is there already a write-up on running this full battery
> of tests? I have a 10 node cluster that I can use for this.
>

There are some instructions.

test/system/continuous/README
test/system/randomwalk/README

Continuous ingest has a lot of options.  For release testing we do
something like the following.

  #configure may need to adjust max mappers and max reducers to make map
reduce job run faster
  start-ingest.sh
  start-walker.sh
  #sleep 24hr
  stop-ingest.sh
  stop-walker.sh
  run-verify.sh

There continuous dir has scripts for starting and stopping the agitator.
 We also use this script to agitate while running random walk test.

For random walk we use the All.xml graph, configure it to log errors to
NFS, and run a walker on each node.  We look in NFS for walkers that died
or got stuck.  The random walk framework will log a message if a node in
the graph gets stuck.  It will also log a message when it gets unstuck.
>
> > Great.  I think this would be a good patch for 1.4.   I assume that if a
> > user stays with Hadoop 1 there are no dependency changes?
>
> Yup. It works the same way as 1.5 where all of the dependency changes
> are in a Hadoop 2.0 profile.
>
> -Joey
>