Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> How to run HDFS from source?


Copy link to this message
-
Re: How to run HDFS from source?
Yes, this should definitely work, though this was briefly broken and fixed
recently. See this JIRA for more info:
https://issues.apache.org/jira/browse/HDFS-2014

--
Aaron T. Myers
Software Engineer, Cloudera

On Thu, Jun 16, 2011 at 7:05 PM, André Oriani <[EMAIL PROTECTED]> wrote:

> I cannot say for the current code, since it look like people had undone the
> project split. But based on Eli`s repo I could, with the code of June 12th,
> do the follow to run HDFS in pseudo distributed mode
>
> -Set  your JAVA_HOME on hadoop-common/conf/hadoop-env.shto match your
> system.
> -Applied
> diff --git a/bin/hadoop b/bin/hadoop
> index 88c49eb..84992e5 100755
> --- a/bin/hadoop
> +++ b/bin/hadoop
> @@ -21,7 +21,7 @@ bin=`which $0`
>  bin=`dirname ${bin}`
>  bin=`cd "$bin"; pwd`
>
> -. "$bin"/../libexec/hadoop-config.sh
> +. "$bin"/hadoop-config.sh
>
>  function print_usage(){
>   echo "Usage: hadoop [--config confdir] COMMAND"
>
>
> - export some environment variables:
> export HADOOP_COMMON_HOME= <hadoop-common`s repo path>
> export HADOOP_HDFS_HOME= <hadoop-hdfs`s repo path>
>
> -Configured core-site.xml and hdfs-site.xml as usual
>
>
> Then $HADOOP_HDFS_HOME/bin/hdfs namenode should work , as well the command
> for datanodes.
>
>
> Personally, I would expect to run from source, because sometimes is good to
> test running the actual thing.
>
>
> Regards,
> André Oriani
>
> On Thu, Jun 16, 2011 at 21:15, Kirk True <[EMAIL PROTECTED]> wrote:
>
> > Should running ./bin/hdfs from the source root work?
> >
> > I get these errors:
> >
> >    [kirk@bubbas apache]$ ./bin/hdfs namenode
> >    ./bin/hdfs: line 154: cygpath: command not found
> >    ./bin/hdfs: line 177: exec: : not found
> >
> > I can hack around the latter by setting the JAVA env var up first:
> >
> >    export JAVA=$JAVA_HOME/bin/java
> >
> > Still, I get this:
> >
> >    [kirk@bubbas apache]$ ./bin/hdfs namenode
> >    ./bin/hdfs: line 154: cygpath: command not found
> >    Exception in thread "main" java.lang.**NoClassDefFoundError:
> > org/apache/hadoop/hdfs/server/**namenode/NameNode
> >    Caused by: java.lang.**ClassNotFoundException:
> > org.apache.hadoop.hdfs.server.**namenode.NameNode
> >        at java.net.URLClassLoader$1.run(**URLClassLoader.java:202)
> >        at java.security.**AccessController.doPrivileged(**Native Method)
> >        at java.net.URLClassLoader.**findClass(URLClassLoader.java:**190)
> >        at java.lang.ClassLoader.**loadClass(ClassLoader.java:**307)
> >        at sun.misc.Launcher$**AppClassLoader.loadClass(**
> > Launcher.java:301)
> >        at java.lang.ClassLoader.**loadClass(ClassLoader.java:**248)
> >    Could not find the main class:
> org.apache.hadoop.hdfs.server.**namenode.NameNode.
> >  Program will exit.
> >
> > This is after running a build (`ant compile`) on the source.
> >
> > I can dig into it, but I'm wondering if running from source/stand-alone
> > like this is even expected to work.
> >
> > -- Kirk
> >
> >
> > On 06/07/2011 01:40 PM, André Oriani wrote:
> >
> >> Thanks a lot Eli, it really helped a lot.  I think I got the general
> idea
> >> of
> >> the scripts.
> >>
> >> Thanks,
> >> André
> >>
> >> On Tue, Jun 7, 2011 at 16:51, Eli Collins<[EMAIL PROTECTED]>  wrote:
> >>
> >>  Hey Andre,
> >>>
> >>> You can run an hdfs build out of common/hdfs trees checked out from
> >>> svn or git. Here are some scripts that make this easier:
> >>>
> >>> https://github.com/elicollins/**hadoop-dev<
> https://github.com/elicollins/hadoop-dev>
> >>>
> >>> Thanks,
> >>> Eli
> >>>
> >>> On Tue, Jun 7, 2011 at 11:56 AM, André Oriani
> >>> <[EMAIL PROTECTED]amp.**br <[EMAIL PROTECTED]>>
> >>>  wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>>
> >>>> I have clone the repo for hadoop-common and hadoop-hdfs and built it
> >>>>
> >>> using
> >>>
> >>>> "ant mvn-install" . Now I would like to be able run HDFS in
> >>>> pseudo-distributed mode  to test some modifications of mine. One year
> >>>> ago
> >>>>
> >>> I
> >>>
> >>>> could do it but now  I had no luck. The scripts are failing,
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB