Hi folks. I was considering sharing some ideas / scripts about how we use
bigtop to test stuff either as code (i.e shell script which invokes various
maven tasks and copies files) to an examples/ folder in the main repo, or
else, write a bigtop blog post (i.e. that walks through the script and
explains the tasks).
Not sure if its worth an "official" code commit, or an official blog post
or wether I should just dump it on my personal blog and move on ... But
let me know ! :)... Here is the basic contents of what the blog post or
example script will walk through.
0) Exports environmental variables
1) builds test-artifacts using standard maven online snapshot mode to get
dependencies and stuff.
2) then rebuilds the artifacts in offline mode to overlay our test-artifact
customizations on top of the work done in the previous step.
3) cp's some pom.xml files into the test-execution/test-artifacts folders,
including a custom iTest groovy snippet to run some pig shell tests.
4) starts flume
5) runs the test-execution task
Overall, it took a while for me to get it fine tuned to be a good cluster
test, but now I'm quite happy with it.
I think it could be a useful artifact to anyone that wants to run bigtop
smoke tests against a non standard hadoop deployment (i.e. anyone like me
... :) )
I'm leaning towards sharing this as a blog post since the script is built
to run against a heavily customized bigtop fork of mine (
github.com/jayunit100/bigtop) which is quite raw and needs some cleanup....
but if someone in the bigtop community wants to collaborate, maybe we can
streamline the example script so that people can use it as a template for
customizing bigtop smokes to their own needs.
Another option, possibly, is we could have a google hangout in the bigtop
community where we share the way we use bigtop to do different tasks and
provide each other with feedback.