Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Request for a pointer to a MapReduce Program tutorial


Copy link to this message
-
Re: Request for a pointer to a MapReduce Program tutorial
Sitamaran,

That was my mistake, you are right it should be "datanode" at the end of
the path. The full path should be:
/home/user/hadoop/data2.2/hdfs/datanode. Sorry about that.
2013/12/28 Sitaraman Vilayannur <[EMAIL PROTECTED]>

> Hi Diego,
>  You have mentioned where my datanode directory is:
> /home/user/hadoop/data2.2/hdfs/namenode.
> it should be datanode ?
> Sitaraman
>
> On 12/29/13, Sitaraman Vilayannur <[EMAIL PROTECTED]> wrote:
> > Thanks Diego, will use the copyToLocal.
> > Sitaraman
> >
> > On 12/29/13, Diego Gutierrez <[EMAIL PROTECTED]> wrote:
> >> Hi,
> >>
> >> For the first question, the name of the file containing the response of
> >> MR
> >> process is like that:  part-r-00000. This file is inside the output
> >> directory.
> >>
> >> The second question, to locate the file in your linux file system you
> >> have
> >> to go to your datanode directory. This directory path is in your
> >> hdfs-site.xml file in {HADOOP_INSTALLATION}/etc/hadoop. In my case the
> >> full
> >> path of the MR response file is:
> >>
> /home/user/hadoop/data2.2/hdfs/datanode/current/BP-1422783414-127.0.1.1-1387647958374/current/finalized.
> >>
> >> where my datanode directory is: /home/user/hadoop/data2.2/hdfs/namenode.
> >>
> >> In the /finalized directory, there are files named like that:
> >> blk_1073741836. In one of those files is the MR response file. This
> >> aproach
> >> is not good if you want to open the response with vi for example,
> instead
> >> you can copy the response file into your linux file system using
> >> *copyToLocal* command(
> >>
> https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/FileSystemShell.html#copyToLocal
> ).
> >> The syntax could be like that: bin/hadoop dfs -copyToLocal
> >> /output/part-r-00000 /home/user/
> >>
> >> Hope this help
> >>
> >>
> >> 2013/12/28 Sitaraman Vilayannur <[EMAIL PROTECTED]>
> >>
> >>> Hi all,
> >>>   Thanks much for the pointers. I ran word count it appears,
> >>> successfully. Currently in inspected the output like so,
> >>> /usr/local/Software/hadoop-2.2.0/bin/hadoop fs -cat
> >>> /user/sitaraman/output/*
> >>> Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library
> >>> /usr/local/Software/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which
> >>> might have disabled stack guard. The VM will try to fix the stack
> >>> guard now.
> >>> It's highly recommended that you fix the library with 'execstack -c
> >>> <libfile>', or link it with '-z noexecstack'.
> >>> 13/12/28 15:13:06 WARN util.NativeCodeLoader: Unable to load
> >>> native-hadoop library for your platform... using builtin-java classes
> >>> where applicable
> >>> as      2
> >>> do      1
> >>> do:     1
> >>> i       2
> >>> not     1
> >>> say     1
> >>>
> >>> What is the name of the file in the output directory?  Can i read it
> >>> say using emacs or vi where will the directory /user/sitaraman/output
> >>> be in the linux fs, that is how to locate where it is....
> >>> Thanks again for all the help.
> >>> Sitaraman
> >>>
> >>> On 12/28/13, Hardik Pandya <[EMAIL PROTECTED]> wrote:
> >>> > I recently blogged about it - hope it helps
> >>> >
> >>> >
> >>>
> http://letsdobigdata.wordpress.com/2013/12/07/running-hadoop-mapreduce-application-from-eclipse-kepler/
> >>> >
> >>> > Regards,
> >>> > Hardik
> >>> >
> >>> > On Fri, Dec 27, 2013 at 6:53 AM, Sitaraman Vilayannur <
> >>> > [EMAIL PROTECTED]> wrote:
> >>> >
> >>> >> Hi,
> >>> >>  Would much appreciate a pointer to a mapreduce tutorial which
> >>> >> explains
> >>> >> how i can run a simulated cluster of mapreduce nodes on a single PC
> >>> >> and
> >>> >> write a Java program with the MapReduce Paradigm.
> >>> >>  Thanks very much.
> >>> >> Sitaraman
> >>> >>
> >>> >
> >>>
> >>
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB