Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> state of HOD


Copy link to this message
-
Re: state of HOD
It has been in a quasi-defunct state for a while now. It seems like
hadoop.next and yarn, helps archive a similar effect of hod. Plus it
has this new hotness factor.

On Fri, Mar 9, 2012 at 2:41 AM, Stijn De Weirdt <[EMAIL PROTECTED]> wrote:
> (my apologies for those who have received this already. i posted this mail a
> few days back on the common-dev list, as this is more a development related
> mail; but one of the original authors/maintainers suggested to also post
> this here)
>
> hi all,
>
> i am a system administrator/user support person/... for the HPC team at
> Ghent University (Ghent, Flanders, Belgium).
>
> recently we have been asked to look into support for hadoop. for the moment
> we are holding off on a dedicated cluster (esp dedicated hdfs setup).
>
> but as all our systems are torque/pbs based, we looked into HOD to help out
> our users.
> we have started from the HOD code that was part of the hadoop 1.0.0 release
> (in the contrib part).
> at first it was not working, but we have been patching and cleaning up the
> code for a a few weeks and now have a version that works for us (we had to
> add some features besides fixing a few things).
> it looks sufficient for now, although we will add some more features soon to
> get the users started.
>
>
> my question is the following: what is the state of HOD atm? is it still
> maintained/supported? are there forks somewhere that have more up-to-date
> code?
> what we are now missing most is the documentation (eg
> http://hadoop.apache.org/common/docs/r0.16.4/hod.html) so we can update this
> with our extra features. is the source available somewhere?
>
> i could contribute back all patches, but a few of them are identation fixes
> (to use 4 space indentation throughout the code) and other cosmetic changes,
> so this messes up patches a lot.
> i have also shuffled a bit with the options (rename and/or move to other
> sections) so no 100% backwards compatibility with the current HOD code.
>
> current main improvements:
> - works with python 2.5 and up (we have been testing with 2.7.2)
> - set options through environment variables
> - better default values (we can now run with empty hodrc file)
> - support for mail and nodes:ppn for pbs
> - no deprecation warnings from hadoop (nearly finished)
> - host-mask to bind xrs addr on non-default ip (in case you have
> non-standard network on the compute nodes)
> - more debug statements
> - gradual code cleanup (using pylint)
>
> on the todo list:
> - further tuning of hadoop parameters (i'm not a hadoop user myself, so this
> will take some time)
> - 0.23.X support
>
>
>
> many thanks,
>
> stijn