Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: Request for a pointer to a MapReduce Program tutorial


Copy link to this message
-
Re: Request for a pointer to a MapReduce Program tutorial
Sitaraman Vilayannur 2013-12-29, 00:47
Hi Diego,
 You have mentioned where my datanode directory is:
/home/user/hadoop/data2.2/hdfs/namenode.
it should be datanode ?
Sitaraman

On 12/29/13, Sitaraman Vilayannur <[EMAIL PROTECTED]> wrote:
> Thanks Diego, will use the copyToLocal.
> Sitaraman
>
> On 12/29/13, Diego Gutierrez <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> For the first question, the name of the file containing the response of
>> MR
>> process is like that:  part-r-00000. This file is inside the output
>> directory.
>>
>> The second question, to locate the file in your linux file system you
>> have
>> to go to your datanode directory. This directory path is in your
>> hdfs-site.xml file in {HADOOP_INSTALLATION}/etc/hadoop. In my case the
>> full
>> path of the MR response file is:
>> /home/user/hadoop/data2.2/hdfs/datanode/current/BP-1422783414-127.0.1.1-1387647958374/current/finalized.
>>
>> where my datanode directory is: /home/user/hadoop/data2.2/hdfs/namenode.
>>
>> In the /finalized directory, there are files named like that:
>> blk_1073741836. In one of those files is the MR response file. This
>> aproach
>> is not good if you want to open the response with vi for example, instead
>> you can copy the response file into your linux file system using
>> *copyToLocal* command(
>> https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/FileSystemShell.html#copyToLocal).
>> The syntax could be like that: bin/hadoop dfs -copyToLocal
>> /output/part-r-00000 /home/user/
>>
>> Hope this help
>>
>>
>> 2013/12/28 Sitaraman Vilayannur <[EMAIL PROTECTED]>
>>
>>> Hi all,
>>>   Thanks much for the pointers. I ran word count it appears,
>>> successfully. Currently in inspected the output like so,
>>> /usr/local/Software/hadoop-2.2.0/bin/hadoop fs -cat
>>> /user/sitaraman/output/*
>>> Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library
>>> /usr/local/Software/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which
>>> might have disabled stack guard. The VM will try to fix the stack
>>> guard now.
>>> It's highly recommended that you fix the library with 'execstack -c
>>> <libfile>', or link it with '-z noexecstack'.
>>> 13/12/28 15:13:06 WARN util.NativeCodeLoader: Unable to load
>>> native-hadoop library for your platform... using builtin-java classes
>>> where applicable
>>> as      2
>>> do      1
>>> do:     1
>>> i       2
>>> not     1
>>> say     1
>>>
>>> What is the name of the file in the output directory?  Can i read it
>>> say using emacs or vi where will the directory /user/sitaraman/output
>>> be in the linux fs, that is how to locate where it is....
>>> Thanks again for all the help.
>>> Sitaraman
>>>
>>> On 12/28/13, Hardik Pandya <[EMAIL PROTECTED]> wrote:
>>> > I recently blogged about it - hope it helps
>>> >
>>> >
>>> http://letsdobigdata.wordpress.com/2013/12/07/running-hadoop-mapreduce-application-from-eclipse-kepler/
>>> >
>>> > Regards,
>>> > Hardik
>>> >
>>> > On Fri, Dec 27, 2013 at 6:53 AM, Sitaraman Vilayannur <
>>> > [EMAIL PROTECTED]> wrote:
>>> >
>>> >> Hi,
>>> >>  Would much appreciate a pointer to a mapreduce tutorial which
>>> >> explains
>>> >> how i can run a simulated cluster of mapreduce nodes on a single PC
>>> >> and
>>> >> write a Java program with the MapReduce Paradigm.
>>> >>  Thanks very much.
>>> >> Sitaraman
>>> >>
>>> >
>>>
>>
>