Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Request for a pointer to a MapReduce Program tutorial


Copy link to this message
-
Re: Request for a pointer to a MapReduce Program tutorial
Sitamaran,

That was my mistake, you are right it should be "datanode" at the end of
the path. The full path should be:
/home/user/hadoop/data2.2/hdfs/datanode. Sorry about that.
2013/12/28 Sitaraman Vilayannur <[EMAIL PROTECTED]>

> Hi Diego,
>  You have mentioned where my datanode directory is:
> /home/user/hadoop/data2.2/hdfs/namenode.
> it should be datanode ?
> Sitaraman
>
> On 12/29/13, Sitaraman Vilayannur <[EMAIL PROTECTED]> wrote:
> > Thanks Diego, will use the copyToLocal.
> > Sitaraman
> >
> > On 12/29/13, Diego Gutierrez <[EMAIL PROTECTED]> wrote:
> >> Hi,
> >>
> >> For the first question, the name of the file containing the response of
> >> MR
> >> process is like that:  part-r-00000. This file is inside the output
> >> directory.
> >>
> >> The second question, to locate the file in your linux file system you
> >> have
> >> to go to your datanode directory. This directory path is in your
> >> hdfs-site.xml file in {HADOOP_INSTALLATION}/etc/hadoop. In my case the
> >> full
> >> path of the MR response file is:
> >>
> /home/user/hadoop/data2.2/hdfs/datanode/current/BP-1422783414-127.0.1.1-1387647958374/current/finalized.
> >>
> >> where my datanode directory is: /home/user/hadoop/data2.2/hdfs/namenode.
> >>
> >> In the /finalized directory, there are files named like that:
> >> blk_1073741836. In one of those files is the MR response file. This
> >> aproach
> >> is not good if you want to open the response with vi for example,
> instead
> >> you can copy the response file into your linux file system using
> >> *copyToLocal* command(
> >>
> https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/FileSystemShell.html#copyToLocal
> ).
> >> The syntax could be like that: bin/hadoop dfs -copyToLocal
> >> /output/part-r-00000 /home/user/
> >>
> >> Hope this help
> >>
> >>
> >> 2013/12/28 Sitaraman Vilayannur <[EMAIL PROTECTED]>
> >>
> >>> Hi all,
> >>>   Thanks much for the pointers. I ran word count it appears,
> >>> successfully. Currently in inspected the output like so,
> >>> /usr/local/Software/hadoop-2.2.0/bin/hadoop fs -cat
> >>> /user/sitaraman/output/*
> >>> Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library
> >>> /usr/local/Software/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which
> >>> might have disabled stack guard. The VM will try to fix the stack
> >>> guard now.
> >>> It's highly recommended that you fix the library with 'execstack -c
> >>> <libfile>', or link it with '-z noexecstack'.
> >>> 13/12/28 15:13:06 WARN util.NativeCodeLoader: Unable to load
> >>> native-hadoop library for your platform... using builtin-java classes
> >>> where applicable
> >>> as      2
> >>> do      1
> >>> do:     1
> >>> i       2
> >>> not     1
> >>> say     1
> >>>
> >>> What is the name of the file in the output directory?  Can i read it
> >>> say using emacs or vi where will the directory /user/sitaraman/output
> >>> be in the linux fs, that is how to locate where it is....
> >>> Thanks again for all the help.
> >>> Sitaraman
> >>>
> >>> On 12/28/13, Hardik Pandya <[EMAIL PROTECTED]> wrote:
> >>> > I recently blogged about it - hope it helps
> >>> >
> >>> >
> >>>
> http://letsdobigdata.wordpress.com/2013/12/07/running-hadoop-mapreduce-application-from-eclipse-kepler/
> >>> >
> >>> > Regards,
> >>> > Hardik
> >>> >
> >>> > On Fri, Dec 27, 2013 at 6:53 AM, Sitaraman Vilayannur <
> >>> > [EMAIL PROTECTED]> wrote:
> >>> >
> >>> >> Hi,
> >>> >>  Would much appreciate a pointer to a mapreduce tutorial which
> >>> >> explains
> >>> >> how i can run a simulated cluster of mapreduce nodes on a single PC
> >>> >> and
> >>> >> write a Java program with the MapReduce Paradigm.
> >>> >>  Thanks very much.
> >>> >> Sitaraman
> >>> >>
> >>> >
> >>>
> >>
> >
>