Hi,

I just setup 3 node hadoop cluster using the latest version from website ,
0.21.0

I am able to start all the daemons, when I run jps I see datanode, namenode,
secondary, tasktracker, but I was running a test and trying to run the
following command: ./hadoop dfs -ls, and I get the following error:

DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/fs/FsShell
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FsShell
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: org.apache.hadoop.fs.FsShell.  Program will
exit.

If i try this command instead:

./hadoop hdfs -ls
Exception in thread "main" java.lang.NoClassDefFoundError: hdfs
Caused by: java.lang.ClassNotFoundException: hdfs
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: hdfs.  Program will exit.

Does anyone know what the command really is I should be using?

Thanks

Hi Mike,

What do you get if you type "./hadoop classpath"? Does it contain the
Hadoop common JAR?

To avoid the deprecation warning you should use "hadoop fs", not "hadoop dfs".

Tom

On Wed, Sep 15, 2010 at 12:53 PM, Mike Franon <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I just setup 3 node hadoop cluster using the latest version from website ,
> 0.21.0
>
> I am able to start all the daemons, when I run jps I see datanode, namenode,
> secondary, tasktracker, but I was running a test and trying to run the
> following command: ./hadoop dfs -ls, and I get the following error:
>
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
> Instead use the hdfs command for it.
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/fs/FsShell
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FsShell
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: org.apache.hadoop.fs.FsShell.  Program will
> exit.
>
> If i try this command instead:
>
> ./hadoop hdfs -ls
> Exception in thread "main" java.lang.NoClassDefFoundError: hdfs
> Caused by: java.lang.ClassNotFoundException: hdfs
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: hdfs.  Program will exit.
>
> Does anyone know what the command really is I should be using?
>
> Thanks
>

Hi Tom,

Great that worked, using ./hadoop -fs ls, and when I did classpath it showed
commonjar.

I had another question not sure why it keeps putting files in /tmp, when in
my configs I have it set to something else.  Most of the files begine with
hadoop-hadoop*.pid, and some Jetty folders.

Here are my configs

<property>
  <name>hadoop.tmp.dir</name>
  <value>/usr/local/hadoop/hadoop-${user.name}</value>
  <description>A base for other temporary directories.</description>
</property>
<property>
  <name>fs.default.name</name>
  <value>hdfs://server1:9000/</value>
</property>
</configuration>
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>dfs.name.dir</name>
  <value>/usr/local/hadoop/hdfs/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/usr/local/hadoop/hdfs/data</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
</configuration>

Thanks,
Mike

On Wed, Sep 15, 2010 at 6:36 PM, Tom White <[EMAIL PROTECTED]> wrote:

> Hi Mike,
>
> What do you get if you type "./hadoop classpath"? Does it contain the
> Hadoop common JAR?
>
> To avoid the deprecation warning you should use "hadoop fs", not "hadoop
> dfs".
>
> Tom
>
> On Wed, Sep 15, 2010 at 12:53 PM, Mike Franon <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > I just setup 3 node hadoop cluster using the latest version from website
> ,
> > 0.21.0
> >
> > I am able to start all the daemons, when I run jps I see datanode,
> namenode,
> > secondary, tasktracker, but I was running a test and trying to run the
> > following command: ./hadoop dfs -ls, and I get the following error:
> >
> > DEPRECATED: Use of this script to execute hdfs command is deprecated.
> > Instead use the hdfs command for it.
> >
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > org/apache/hadoop/fs/FsShell
> > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FsShell
> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > Could not find the main class: org.apache.hadoop.fs.FsShell.  Program
> will
> > exit.
> >
> > If i try this command instead:
> >
> > ./hadoop hdfs -ls
> > Exception in thread "main" java.lang.NoClassDefFoundError: hdfs
> > Caused by: java.lang.ClassNotFoundException: hdfs
> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > Could not find the main class: hdfs.  Program will exit.
> >
> > Does anyone know what the command really is I should be using?
> >
> > Thanks
> >
>

You need to set the PID dir separately using HADOOP_PID_DIR  in conf/hadoop-env.sh

-Avani
-----Original Message-----
From: Mike Franon [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 16, 2010 6:33 AM
To: [EMAIL PROTECTED]
Subject: Re: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FsShell

Hi Tom,

Great that worked, using ./hadoop -fs ls, and when I did classpath it showed
commonjar.

I had another question not sure why it keeps putting files in /tmp, when in
my configs I have it set to something else.  Most of the files begine with
hadoop-hadoop*.pid, and some Jetty folders.

Here are my configs

<property>
  <name>hadoop.tmp.dir</name>
  <value>/usr/local/hadoop/hadoop-${user.name}</value>
  <description>A base for other temporary directories.</description>
</property>
<property>
  <name>fs.default.name</name>
  <value>hdfs://server1:9000/</value>
</property>
</configuration>
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>dfs.name.dir</name>
  <value>/usr/local/hadoop/hdfs/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/usr/local/hadoop/hdfs/data</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
</configuration>

Thanks,
Mike

On Wed, Sep 15, 2010 at 6:36 PM, Tom White <[EMAIL PROTECTED]> wrote:

> Hi Mike,
>
> What do you get if you type "./hadoop classpath"? Does it contain the
> Hadoop common JAR?
>
> To avoid the deprecation warning you should use "hadoop fs", not "hadoop
> dfs".
>
> Tom
>
> On Wed, Sep 15, 2010 at 12:53 PM, Mike Franon <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > I just setup 3 node hadoop cluster using the latest version from website
> ,
> > 0.21.0
> >
> > I am able to start all the daemons, when I run jps I see datanode,
> namenode,
> > secondary, tasktracker, but I was running a test and trying to run the
> > following command: ./hadoop dfs -ls, and I get the following error:
> >
> > DEPRECATED: Use of this script to execute hdfs command is deprecated.
> > Instead use the hdfs command for it.
> >
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > org/apache/hadoop/fs/FsShell
> > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FsShell
> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > Could not find the main class: org.apache.hadoop.fs.FsShell.  Program
> will
> > exit.
> >
> > If i try this command instead:
> >
> > ./hadoop hdfs -ls
> > Exception in thread "main" java.lang.NoClassDefFoundError: hdfs
> > Caused by: java.lang.ClassNotFoundException: hdfs
> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > Could not find the main class: hdfs.  Program will exit.
> >
> > Does anyone know what the command really is I should be using?
> >
> > Thanks
> >
>

Great thanks for the help!

On Thu, Sep 16, 2010 at 1:14 PM, Sharma, Avani <[EMAIL PROTECTED]> wrote:

> You need to set the PID dir separately using HADOOP_PID_DIR  in
> conf/hadoop-env.sh
>
> -Avani
> -----Original Message-----
> From: Mike Franon [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, September 16, 2010 6:33 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/fs/FsShell
>
> Hi Tom,
>
> Great that worked, using ./hadoop -fs ls, and when I did classpath it
> showed
> commonjar.
>
> I had another question not sure why it keeps putting files in /tmp, when in
> my configs I have it set to something else.  Most of the files begine with
> hadoop-hadoop*.pid, and some Jetty folders.
>
> Here are my configs
>
> <property>
>  <name>hadoop.tmp.dir</name>
>  <value>/usr/local/hadoop/hadoop-${user.name}</value>
>  <description>A base for other temporary directories.</description>
> </property>
>
>
> <property>
>  <name>fs.default.name</name>
>  <value>hdfs://server1:9000/</value>
> </property>
> </configuration>
>
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
>  <name>dfs.name.dir</name>
>  <value>/usr/local/hadoop/hdfs/name</value>
> </property>
> <property>
>  <name>dfs.data.dir</name>
>  <value>/usr/local/hadoop/hdfs/data</value>
> </property>
> <property>
>  <name>dfs.replication</name>
>  <value>3</value>
> </property>
> </configuration>
>
> Thanks,
> Mike
>
> On Wed, Sep 15, 2010 at 6:36 PM, Tom White <[EMAIL PROTECTED]> wrote:
>
> > Hi Mike,
> >
> > What do you get if you type "./hadoop classpath"? Does it contain the
> > Hadoop common JAR?
> >
> > To avoid the deprecation warning you should use "hadoop fs", not "hadoop
> > dfs".
> >
> > Tom
> >
> > On Wed, Sep 15, 2010 at 12:53 PM, Mike Franon <[EMAIL PROTECTED]>
> > wrote:
> > > Hi,
> > >
> > > I just setup 3 node hadoop cluster using the latest version from
> website
> > ,
> > > 0.21.0
> > >
> > > I am able to start all the daemons, when I run jps I see datanode,
> > namenode,
> > > secondary, tasktracker, but I was running a test and trying to run the
> > > following command: ./hadoop dfs -ls, and I get the following error:
> > >
> > > DEPRECATED: Use of this script to execute hdfs command is deprecated.
> > > Instead use the hdfs command for it.
> > >
> > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > org/apache/hadoop/fs/FsShell
> > > Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.fs.FsShell
> > >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > >        at java.security.AccessController.doPrivileged(Native Method)
> > >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > > Could not find the main class: org.apache.hadoop.fs.FsShell.  Program
> > will
> > > exit.
> > >
> > > If i try this command instead:
> > >
> > > ./hadoop hdfs -ls
> > > Exception in thread "main" java.lang.NoClassDefFoundError: hdfs
> > > Caused by: java.lang.ClassNotFoundException: hdfs
> > >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > >        at java.security.AccessController.doPrivileged(Native Method)
> > >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > > Could not find the main class: hdfs.  Program will exit.
> > >
> > > Does anyone know what the command really is I should be using?
> > >
> > > Thanks
> > >
> >
>

Hi,
I know this question has been asked before, but I could not find
the right solution.  Maybe because I use hadoop 0.20.2, some posts assumed older versions.
My code (relevant chunk):
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);

the last line gives:
Exception in thread "main" java.io.IOException: No FileSystem for
scheme: file
I launched this like so:
java -cp /usr/local/hadoop/src/core/:/usr/local/hadoop/conf/ -jar
myjar.jar
AFAICT, this should make sure all the configuration can be found and it should be able to connect to the filesystem:

dplaetin@n-0:~$ ls -alh /usr/local/hadoop/src/core/ /usr/local/hadoop/conf/
/usr/local/hadoop/conf/:
total 64K
drwxr-xr-x  2 dplaetin Search 4.0K Aug 26 17:21 .
drwxr-xr-x 12 root     root   4.0K Feb 19  2010 ..
-rw-rw-r--  1 root     root   3.9K Feb 19  2010 capacity-scheduler.xml
-rw-rw-r--  1 root     root    535 Feb 19  2010 configuration.xsl
-rw-r--r--  1 dplaetin Search  459 Apr 29 15:06 core-site.xml
-rw-r--r--  1 dplaetin Search 2.3K Apr 11 14:23 hadoop-env.sh
-rw-rw-r--  1 root     root   1.3K Feb 19  2010 hadoop-metrics.properties
-rw-rw-r--  1 root     root   4.1K Feb 19  2010 hadoop-policy.xml
-rw-r--r--  1 dplaetin Search  490 Apr 11 10:18 hdfs-site.xml
-rw-r--r--  1 dplaetin Search 2.8K Apr 11 14:23 log4j.properties
-rw-r--r--  1 dplaetin Search 1.1K Aug  3 09:49 mapred-site.xml
-rw-rw-r--  1 root     root     10 Feb 19  2010 masters
-rw-r--r--  1 dplaetin Search   95 Apr  4 17:17 slaves
-rw-rw-r--  1 root     root   1.3K Feb 19  2010 ssl-client.xml.example
-rw-rw-r--  1 root     root   1.2K Feb 19  2010 ssl-server.xml.example

/usr/local/hadoop/src/core/:
total 36K
drwxr-xr-x  3 root root 4.0K Aug 24 10:40 .
drwxr-xr-x 15 root root 4.0K Aug 24 10:40 ..
-rw-rw-r--  1 root root  14K Feb 19  2010 core-default.xml
drwxr-xr-x  3 root root 4.0K Feb 19  2010 org
-rw-rw-r--  1 root root 7.9K Feb 19  2010 overview.html

Specifically, you can see I have a core-default.xml and a core-site.xml, which should be all that's needed,
according to the org.apache.hadoop.conf.Configuration documentation
(http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/conf/Configuration.html)
I read somewhere there should be a hadoop-default.xml; I thought this was deprecated but to be sure I
created the empty file in /usr/local/hadoop/src/core/ , but the error remained the same.

The cluster works fine, I've done tens of jobs on it, but as you can see, something fails when I try to interface to it directly.

Thanks in advance for any help,
Dieter