Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> DFS respond very slow


Copy link to this message
-
Re: DFS respond very slow
Also, note that JVM startup overhead, etc, means your -ls time is not
completely unreasonable. Using OpenJDK on a cluster of VMs, my "hdfs
dfs -ls" takes 1.88 seconds according to time (and 1.59 seconds of
user CPU time).

I'd be much more concerned about your slow transfer times.  On the
same cluster, I can easily push 4 MB/sec even with only a 100MB file
using "hdfs dfs -put - foo.txt". And of course using distcp or
multiple -put workloads HDFS can saturate multiple GigE links.

-andy

On Mon, Oct 15, 2012 at 5:22 PM, Vinod Kumar Vavilapalli
<[EMAIL PROTECTED]> wrote:
> Try picking up a single operation say "hadoop dfs -ls" and start profiling.
>  - Time the client JVM is taking to start. Enable debug logging on the
> client side by exporting HADOOP_ROOT_LOGGER=DEBUG,CONSOLE
>  - Time between the client starting and the namenode audit logs showing the
> read request. Also enable debug logging on the daemons too.
>  - Also, you can wget the namenode web pages and see how fast they return.
>
> To repeat what is already obvious, It is most likely related to your network
> setup and/or configuration.
>
> Thanks,
> +Vinod
>
> On Oct 10, 2012, at 12:20 AM, Alexey wrote:
>
> ok, here you go:
> I have 3 servers:
> datanode on server 1, 2, 3
> namenode on server 1
> secondarynamenode on server 2
>
> all servers are at the hetzner datacenter and connected through 100Mbit
> link, pings between them about 0.1ms
>
> each server has 24Gb ram and intel core i7 3Ghz CPU
> disk is 700Gb RAID
>
> the bindings related configuration is the following:
> server 1:
> core-site.xml
> --------------------------------------
> <name>fs.default.name</name>
> <value>hdfs://5.6.7.11:8020</value>
> --------------------------------------
>
> hdfs-site.xml
> --------------------------------------
> <name>dfs.datanode.address</name>
> <value>0.0.0.0:50010</value>
>
> <name>dfs.datanode.http.address</name>
> <value>0.0.0.0:50075</value>
>
> <name>dfs.http.address</name>
> <value>5.6.7.11:50070</value>
>
> <name>dfs.secondary.https.port</name>
> <value>50490</value>
>
> <name>dfs.https.port</name>
> <value>50470</value>
>
> <name>dfs.https.address</name>
> <value>5.6.7.11:50470</value>
>
> <name>dfs.secondary.http.address</name>
> <value>5.6.7.12:50090</value>
> --------------------------------------
>
> server 2:
> core-site.xml
> --------------------------------------
> <name>fs.default.name</name>
> <value>hdfs://5.6.7.11:8020</value>
> --------------------------------------
>
> hdfs-site.xml
> --------------------------------------
> <name>dfs.datanode.address</name>
> <value>0.0.0.0:50010</value>
>
> <name>dfs.datanode.http.address</name>
> <value>0.0.0.0:50075</value>
>
> <name>dfs.http.address</name>
> <value>5.6.7.11:50070</value>
>
> <name>dfs.secondary.https.port</name>
> <value>50490</value>
>
> <name>dfs.https.port</name>
> <value>50470</value>
>
> <name>dfs.https.address</name>
> <value>5.6.7.11:50470</value>
>
> <name>dfs.secondary.http.address</name>
> <value>5.6.7.12:50090</value>
> --------------------------------------
>
> server 3:
> core-site.xml
> --------------------------------------
> <name>fs.default.name</name>
> <value>hdfs://5.6.7.11:8020</value>
> --------------------------------------
>
> hdfs-site.xml
> --------------------------------------
> <name>dfs.datanode.address</name>
> <value>0.0.0.0:50010</value>
>
> <name>dfs.datanode.http.address</name>
> <value>0.0.0.0:50075</value>
>
> <name>dfs.http.address</name>
> <value>127.0.0.1:50070</value>
>
> <name>dfs.secondary.https.port</name>
> <value>50490</value>
>
> <name>dfs.https.port</name>
> <value>50470</value>
>
> <name>dfs.https.address</name>
> <value>127.0.0.1:50470</value>
>
> <name>dfs.secondary.http.address</name>
> <value>5.6.7.12:50090</value>
> --------------------------------------
>
> netstat output:
> server 1
>
> tcp        0      0 5.6.7.11:8020           0.0.0.0:*               LISTEN
> 10870/java
>
> tcp        0      0 5.6.7.11:50070          0.0.0.0:*               LISTEN
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB