Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> state of HOD


Copy link to this message
-
Re: state of HOD
It has been in a quasi-defunct state for a while now. It seems like
hadoop.next and yarn, helps archive a similar effect of hod. Plus it
has this new hotness factor.

On Fri, Mar 9, 2012 at 2:41 AM, Stijn De Weirdt <[EMAIL PROTECTED]> wrote:
> (my apologies for those who have received this already. i posted this mail a
> few days back on the common-dev list, as this is more a development related
> mail; but one of the original authors/maintainers suggested to also post
> this here)
>
> hi all,
>
> i am a system administrator/user support person/... for the HPC team at
> Ghent University (Ghent, Flanders, Belgium).
>
> recently we have been asked to look into support for hadoop. for the moment
> we are holding off on a dedicated cluster (esp dedicated hdfs setup).
>
> but as all our systems are torque/pbs based, we looked into HOD to help out
> our users.
> we have started from the HOD code that was part of the hadoop 1.0.0 release
> (in the contrib part).
> at first it was not working, but we have been patching and cleaning up the
> code for a a few weeks and now have a version that works for us (we had to
> add some features besides fixing a few things).
> it looks sufficient for now, although we will add some more features soon to
> get the users started.
>
>
> my question is the following: what is the state of HOD atm? is it still
> maintained/supported? are there forks somewhere that have more up-to-date
> code?
> what we are now missing most is the documentation (eg
> http://hadoop.apache.org/common/docs/r0.16.4/hod.html) so we can update this
> with our extra features. is the source available somewhere?
>
> i could contribute back all patches, but a few of them are identation fixes
> (to use 4 space indentation throughout the code) and other cosmetic changes,
> so this messes up patches a lot.
> i have also shuffled a bit with the options (rename and/or move to other
> sections) so no 100% backwards compatibility with the current HOD code.
>
> current main improvements:
> - works with python 2.5 and up (we have been testing with 2.7.2)
> - set options through environment variables
> - better default values (we can now run with empty hodrc file)
> - support for mail and nodes:ppn for pbs
> - no deprecation warnings from hadoop (nearly finished)
> - host-mask to bind xrs addr on non-default ip (in case you have
> non-standard network on the compute nodes)
> - more debug statements
> - gradual code cleanup (using pylint)
>
> on the todo list:
> - further tuning of hadoop parameters (i'm not a hadoop user myself, so this
> will take some time)
> - 0.23.X support
>
>
>
> many thanks,
>
> stijn
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB