Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Problems with HOD and HDFS


Copy link to this message
-
Re: Problems with HOD and HDFS
Hey Dave,

I can't speak for the folks at Yahoo!, but from watching the JIRA, I don't
think HOD is actively used or developed anywhere these days. You're
attempting to use a mostly deprecated project, and hence not receiving any
support on the mailing list.

Thanks,
Jeff

On Sun, Jun 13, 2010 at 7:33 PM, David Milne <[EMAIL PROTECTED]> wrote:

> Anybody? I am completely stuck here. I have no idea who else I can ask
> or where I can go for more information. Is there somewhere specific
> where I should be asking about HOD?
>
> Thank you,
> Dave
>
> On Thu, Jun 10, 2010 at 2:56 PM, David Milne <[EMAIL PROTECTED]> wrote:
> > Hi there,
> >
> > I am trying to get Hadoop on Demand up and running, but am having
> > problems with the ringmaster not being able to communicate with HDFS.
> >
> > The output from the hod allocate command ends with this, with full
> verbosity:
> >
> > [2010-06-10 14:40:22,650] CRITICAL/50 hadoop:298 - Failed to retrieve
> > 'hdfs' service address.
> > [2010-06-10 14:40:22,654] DEBUG/10 hadoop:631 - Cleaning up cluster id
> > 34029.symphony.cs.waikato.ac.nz, as cluster could not be allocated.
> > [2010-06-10 14:40:22,655] DEBUG/10 hadoop:635 - Calling rm.stop()
> > [2010-06-10 14:40:22,665] DEBUG/10 hadoop:637 - Returning from rm.stop()
> > [2010-06-10 14:40:22,666] CRITICAL/50 hod:401 - Cannot allocate
> > cluster /home/dmilne/hadoop/cluster
> > [2010-06-10 14:40:23,090] DEBUG/10 hod:597 - return code: 7
> >
> >
> > I've attached the hodrc file below, but briefly HOD is supposed to
> > provision an HDFS cluster as well as a Map/Reduce cluster, and seems
> > to be failing to do so. The ringmaster log looks like this:
> >
> > [2010-06-10 14:36:05,144] DEBUG/10 ringMaster:479 - getServiceAddr name:
> hdfs
> > [2010-06-10 14:36:05,145] DEBUG/10 ringMaster:487 - getServiceAddr
> > service: <hodlib.GridServices.hdfs.Hdfs instance at 0x8f97e8>
> > [2010-06-10 14:36:05,147] DEBUG/10 ringMaster:504 - getServiceAddr
> > addr hdfs: not found
> > [2010-06-10 14:36:06,195] DEBUG/10 ringMaster:479 - getServiceAddr name:
> hdfs
> > [2010-06-10 14:36:06,197] DEBUG/10 ringMaster:487 - getServiceAddr
> > service: <hodlib.GridServices.hdfs.Hdfs instance at 0x8f97e8>
> > [2010-06-10 14:36:06,198] DEBUG/10 ringMaster:504 - getServiceAddr
> > addr hdfs: not found
> >
> > ... and so on, until it gives up
> >
> > Any ideas why? One red flag is that when running the allocate command,
> > some of the variables echo-ed back look dodgy:
> >
> > --gridservice-hdfs.fs_port 0
> > --gridservice-hdfs.host localhost
> > --gridservice-hdfs.info_port 0
> >
> > These are not what I specified in the hodrc. Are the port numbers just
> > set to 0 because I am not using an external HDFS, or is this a
> > problem?
> >
> >
> > The software versions involved are:
> >  - Hadoop 0.20.2
> >  - Python 2.5.2 (no Twisted)
> >  - Java 1.6.0_20
> >  - Torque 2.4.5
> >
> >
> > The hodrc file looks like this:
> >
> > [hod]
> > stream                          = True
> > java-home                       = /opt/jdk1.6.0_20
> > cluster                         = debian5
> > cluster-factor                  = 1.8
> > xrs-port-range                  = 32768-65536
> > debug                           = 3
> > allocate-wait-time              = 3600
> > temp-dir                        = /scratch/local/dmilne/hod
> >
> > [ringmaster]
> > register                        = True
> > stream                          = False
> > temp-dir                        = /scratch/local/dmilne/hod
> > log-dir                         = /scratch/local/dmilne/hod/log
> > http-port-range                 = 8000-9000
> > idleness-limit                  = 864000
> > work-dirs                       > > /scratch/local/dmilne/hod/1,/scratch/local/dmilne/hod/2
> > xrs-port-range                  = 32768-65536
> > debug                           = 4
> >
> > [hodring]
> > stream                          = False
> > temp-dir                        = /scratch/local/dmilne/hod
> > log-dir                         = /scratch/local/dmilne/hod/log
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB