Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Monitor network traffic in hadoop


Copy link to this message
-
Re: Monitor network traffic in hadoop
Marco Shaw 2013-12-08, 22:32
Try something like:
http://tutorialgenius.blogspot.ca/2012/03/tcpdump-localhost.html

After you launch tcpdump it mentions "eth0", while your mapred-site.xml is
referring to "localhost" or "lo".  Perhaps...

(Sorry, I don't have a VM running and can't remember if it's "lo" "elle-o".)
On Sun, Dec 8, 2013 at 2:57 PM, navaz <[EMAIL PROTECTED]> wrote:

> Thanks. I have tried with sudo command and command executed successfully.
> But nothing is captured.
>
> hduser@ubuntu:~/Downloads$ sudo tcpdump port 13562
> [sudo] password for hduser:
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
>
> I am running the mapreduce program and this is the output.
>
> hduser@ubuntu:~/Downloads$ hadoop jar chiu-wordcount2.jar WordCount
> /user/hduser/getty/gettysburg.txt /user/hduser/getty/out8
> Warning: $HADOOP_HOME is deprecated.
>
> 13/12/08 10:43:47 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 13/12/08 10:43:47 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/12/08 10:43:47 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/12/08 10:43:47 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/12/08 10:43:48 INFO mapred.JobClient: Running job: job_201312081040_0001
> 13/12/08 10:43:49 INFO mapred.JobClient:  map 0% reduce 0%
> 13/12/08 10:43:54 INFO mapred.JobClient:  map 100% reduce 0%
> 13/12/08 10:44:02 INFO mapred.JobClient:  map 100% reduce 33%
> 13/12/08 10:44:03 INFO mapred.JobClient:  map 100% reduce 100%
> 13/12/08 10:44:04 INFO mapred.JobClient: Job complete:
> job_201312081040_0001
> 13/12/08 10:44:04 INFO mapred.JobClient: Counters: 29
> 13/12/08 10:44:04 INFO mapred.JobClient:   Job Counters
> 13/12/08 10:44:04 INFO mapred.JobClient:     Launched reduce tasks=1
> 13/12/08 10:44:04 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=5478
> 13/12/08 10:44:04 INFO mapred.JobClient:     Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/12/08 10:44:04 INFO mapred.JobClient:     Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/12/08 10:44:04 INFO mapred.JobClient:     Launched map tasks=1
> 13/12/08 10:44:04 INFO mapred.JobClient:     Data-local map tasks=1
> 13/12/08 10:44:04 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=8804
> 13/12/08 10:44:04 INFO mapred.JobClient:   File Output Format Counters
> 13/12/08 10:44:04 INFO mapred.JobClient:     Bytes Written=1106
> 13/12/08 10:44:04 INFO mapred.JobClient:   FileSystemCounters
> 13/12/08 10:44:04 INFO mapred.JobClient:     FILE_BYTES_READ=3059
> 13/12/08 10:44:04 INFO mapred.JobClient:     HDFS_BYTES_READ=1606
> 13/12/08 10:44:04 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=108770
> 13/12/08 10:44:04 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=1106
> 13/12/08 10:44:04 INFO mapred.JobClient:   File Input Format Counters
> 13/12/08 10:44:04 INFO mapred.JobClient:     Bytes Read=1486
> 13/12/08 10:44:04 INFO mapred.JobClient:   Map-Reduce Framework
> 13/12/08 10:44:04 INFO mapred.JobClient:     Map output materialized
> bytes=3059
> 13/12/08 10:44:04 INFO mapred.JobClient:     Map input records=6
> 13/12/08 10:44:04 INFO mapred.JobClient:     Reduce shuffle bytes=3059
> 13/12/08 10:44:04 INFO mapred.JobClient:     Spilled Records=544
> 13/12/08 10:44:04 INFO mapred.JobClient:     Map output bytes=2509
> 13/12/08 10:44:04 INFO mapred.JobClient:     Total committed heap usage
> (bytes)=131600384
> 13/12/08 10:44:04 INFO mapred.JobClient:     CPU time spent (ms)=1010
> 13/12/08 10:44:04 INFO mapred.JobClient:     Combine input records=0
> 13/12/08 10:44:04 INFO mapred.JobClient:     SPLIT_RAW_BYTES=120
> 13/12/08 10:44:04 INFO mapred.JobClient:     Reduce input records=272
> 13/12/08 10:44:04 INFO mapred.JobClient:     Reduce input groups=138
> 13/12/08 10:44:04 INFO mapred.JobClient:     Combine output records=0
> 13/12/08 10:44:04 INFO mapred.JobClient:     Physical memory (bytes)