Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - wordcount example using local file system instead of distributed one?


Copy link to this message
-
Re: wordcount example using local file system instead of distributed one?
Aaron Eng 2010-12-08, 23:30
>Why did that work out of curiosity?
Each of the config files can be used to set only specific configuration
values.  So even though you had a valid config property/value, it didn't
apply because it wasn't in the right file.  So, why did it work? Because
thats the way it is coded.  A better question, why didn't it work with your
original config files? Because of poor usability.  I think you can find a
link to the config values that work in the various config files somewhere on
the Apache Hadoop site (that was vague, wasn't it?).

>Am I doing something wrong
Yes, you need to run start-dfs.sh on the node that you want to become the
namenode and you need to run start-mapred.sh on the node that you want to
become jobtracker.  Again, the reason is very poor usability and bad
scripting.  Of course, someone will inevitably say that you could just right
your own scripts to control the services...
On Wed, Dec 8, 2010 at 3:09 PM, Hiller, Dean (Contractor) <
[EMAIL PROTECTED]> wrote:

> Sweeeeeeet!!!!! That worked…didn’t see that in the docs at all.  Why did
> that work out of curiosity?
>
>
>
> Also, I run ./start-dfs.sh on node 1 and ./start-mapred.sh on node 2.  Am I
> doing something wrong in that
>
> I cannot run those on any other nodes L.  Ie. If I run it on a different
> node, the NameNode process does not
>
> Run on node 1 and the JobTracker does not run on node 2.  It is like those
> are locked to those boxes because
>
> Of the config where the slaves file allow all the slaves to be started just
> fine.  Am I doing something wrong or
>
> Is that just simply always the case?
>
>
>
> My config is mapred is set to jobtracker of node 2, and my masters only has
> node 1 and my slaves has node 1
>
> And node 2.  Is there no mapred masters like file for when I run
> ./start-mapred?
>
>
>
> I would think ./start-dfs.sh should work from any node since masters file
> and slaves file contain all the nodes they
>
> Need to start things on, but it doesn’t seem to work?  (just want it to be
> more seamless in case I am on the wrong node
>
> as right now, it seems to only shut down some things and not all if done
> from wrong node).
>
>
>
> Thanks,
>
> Dean
>
>
>
> *From:* Aaron Eng [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, December 08, 2010 3:57 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: wordcount example using local file system instead of
> distributed one?
>
>
>
> You will also need to restart services after that, in case that wasn't
> obvious.
>
> On Wed, Dec 8, 2010 at 2:56 PM, Aaron Eng <[EMAIL PROTECTED]> wrote:
>
> Hi Dean,
>
> Try removing the fs.default.name parameter from hdfs-site.xml and put it
> in core-site.xml
>
>
>
> On Wed, Dec 8, 2010 at 2:46 PM, Hiller, Dean (Contractor) <
> [EMAIL PROTECTED]> wrote:
>
> I run the following wordcount example(my hadoop shell seems to always hit
> the local file system first so I had to add the hdfs…is that normal??...I
> mean, I see it printing configDir= which is where I moved the config dir and
> what I set the env var too which has the location in the config files there
> but it still hits the local).
>
>
>
> [root@localhost hadoop]# ./bin/hadoop jar hadoop-0.20.2-examples.jar
> wordcount
>
> hdfs://206.88.43.8:54310/wordcount hdfs://
> 206.88.43.168:54310/wordcount-out
>
>
>
> configDir=/mnt/mucho/hadoop-config/
>
>
> classpath=/opt/hbase-install/hbase/hbase-0.20.6.jar:/opt/hbase-install/hbase/hba
>
>
> se-0.20.6-test.jar:/mnt/mucho/hbase-config/:/opt/hbase-install/hbase/lib/zookeep
>
> er-3.2.2.jar
>
> 10/12/08 08:42:33 INFO input.FileInputFormat: Total input paths to process
> : 13
>
> org.apache.hadoop.ipc.RemoteException: java.io.FileNotFoundException: File
> file:
>
> /tmp/hadoop-root/mapred/system/job_201012080654_0010/job.xml does not
> exist.
>
>         at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSys
>
> tem.java:361)
>
>         at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.
>
> java:245)