Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> wordcount example using local file system instead of distributed one?


Copy link to this message
-
Re: wordcount example using local file system instead of distributed one?
You will also need to restart services after that, in case that wasn't
obvious.

On Wed, Dec 8, 2010 at 2:56 PM, Aaron Eng <[EMAIL PROTECTED]> wrote:

> Hi Dean,
>
> Try removing the fs.default.name parameter from hdfs-site.xml and put it
> in core-site.xml
>
>
>
> On Wed, Dec 8, 2010 at 2:46 PM, Hiller, Dean (Contractor) <
> [EMAIL PROTECTED]> wrote:
>
>> I run the following wordcount example(my hadoop shell seems to always hit
>> the local file system first so I had to add the hdfs…is that normal??...I
>> mean, I see it printing configDir= which is where I moved the config dir and
>> what I set the env var too which has the location in the config files there
>> but it still hits the local).
>>
>>
>>
>> [root@localhost hadoop]# ./bin/hadoop jar hadoop-0.20.2-examples.jar
>> wordcount
>>
>> hdfs://206.88.43.8:54310/wordcount hdfs://
>> 206.88.43.168:54310/wordcount-out
>>
>>
>>
>> configDir=/mnt/mucho/hadoop-config/
>>
>>
>> classpath=/opt/hbase-install/hbase/hbase-0.20.6.jar:/opt/hbase-install/hbase/hba
>>
>>
>> se-0.20.6-test.jar:/mnt/mucho/hbase-config/:/opt/hbase-install/hbase/lib/zookeep
>>
>> er-3.2.2.jar
>>
>> 10/12/08 08:42:33 INFO input.FileInputFormat: Total input paths to process
>> : 13
>>
>> org.apache.hadoop.ipc.RemoteException: java.io.FileNotFoundException: File
>> file:
>>
>> /tmp/hadoop-root/mapred/system/job_201012080654_0010/job.xml does not
>> exist.
>>
>>         at
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSys
>>
>> tem.java:361)
>>
>>         at
>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.
>>
>> java:245)
>>
>>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
>>
>>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
>>
>>         at
>> org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.
>>
>> java:61)
>>
>>         at
>> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)
>>
>>
>>
>>         at
>> org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:257)
>>
>>
>>
>>         at
>> org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:234)
>>
>>
>>
>>         at
>> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2993)
>>
>>
>>
>> In case it helps, here is my hdfs-site.xml that is used by the daemons
>> started AND the client(is that an issue…using the same one)…
>>
>> <configuration>
>>
>> <property>
>>
>>   <name>fs.default.name</name>
>>
>>   <value>hdfs://206.88.43.168:54310</value>
>>
>> </property>
>>
>> <property>
>>
>>   <name>hadoop.tmp.dir</name>
>>
>>   <value>/opt/data/hadooptmp</value>
>>
>> </property>
>>
>> <property>
>>
>>   <name>dfs.data.dir</name>
>>
>>   <value>/opt/data/hadoop</value>
>>
>> </property>
>>
>> <property>
>>
>>   <name>dfs.replication</name>
>>
>>   <value>2</value>
>>
>> </property>
>>
>> </configuration>
>>
>>
>>
>>
>>
>> This message and any attachments are intended only for the use of the addressee and
>> may contain information that is privileged and confidential. If the reader of the
>> message is not the intended recipient or an authorized representative of the
>> intended recipient, you are hereby notified that any dissemination of this
>> communication is strictly prohibited. If you have received this communication in
>> error, please notify us immediately by e-mail and delete the message and any
>> attachments from your system.
>>
>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB