Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> DFS respond very slow


Copy link to this message
-
Re: DFS respond very slow
Also, note that JVM startup overhead, etc, means your -ls time is not
completely unreasonable. Using OpenJDK on a cluster of VMs, my "hdfs
dfs -ls" takes 1.88 seconds according to time (and 1.59 seconds of
user CPU time).

I'd be much more concerned about your slow transfer times.  On the
same cluster, I can easily push 4 MB/sec even with only a 100MB file
using "hdfs dfs -put - foo.txt". And of course using distcp or
multiple -put workloads HDFS can saturate multiple GigE links.

-andy

On Mon, Oct 15, 2012 at 5:22 PM, Vinod Kumar Vavilapalli
<[EMAIL PROTECTED]> wrote:
> Try picking up a single operation say "hadoop dfs -ls" and start profiling.
>  - Time the client JVM is taking to start. Enable debug logging on the
> client side by exporting HADOOP_ROOT_LOGGER=DEBUG,CONSOLE
>  - Time between the client starting and the namenode audit logs showing the
> read request. Also enable debug logging on the daemons too.
>  - Also, you can wget the namenode web pages and see how fast they return.
>
> To repeat what is already obvious, It is most likely related to your network
> setup and/or configuration.
>
> Thanks,
> +Vinod
>
> On Oct 10, 2012, at 12:20 AM, Alexey wrote:
>
> ok, here you go:
> I have 3 servers:
> datanode on server 1, 2, 3
> namenode on server 1
> secondarynamenode on server 2
>
> all servers are at the hetzner datacenter and connected through 100Mbit
> link, pings between them about 0.1ms
>
> each server has 24Gb ram and intel core i7 3Ghz CPU
> disk is 700Gb RAID
>
> the bindings related configuration is the following:
> server 1:
> core-site.xml
> --------------------------------------
> <name>fs.default.name</name>
> <value>hdfs://5.6.7.11:8020</value>
> --------------------------------------
>
> hdfs-site.xml
> --------------------------------------
> <name>dfs.datanode.address</name>
> <value>0.0.0.0:50010</value>
>
> <name>dfs.datanode.http.address</name>
> <value>0.0.0.0:50075</value>
>
> <name>dfs.http.address</name>
> <value>5.6.7.11:50070</value>
>
> <name>dfs.secondary.https.port</name>
> <value>50490</value>
>
> <name>dfs.https.port</name>
> <value>50470</value>
>
> <name>dfs.https.address</name>
> <value>5.6.7.11:50470</value>
>
> <name>dfs.secondary.http.address</name>
> <value>5.6.7.12:50090</value>
> --------------------------------------
>
> server 2:
> core-site.xml
> --------------------------------------
> <name>fs.default.name</name>
> <value>hdfs://5.6.7.11:8020</value>
> --------------------------------------
>
> hdfs-site.xml
> --------------------------------------
> <name>dfs.datanode.address</name>
> <value>0.0.0.0:50010</value>
>
> <name>dfs.datanode.http.address</name>
> <value>0.0.0.0:50075</value>
>
> <name>dfs.http.address</name>
> <value>5.6.7.11:50070</value>
>
> <name>dfs.secondary.https.port</name>
> <value>50490</value>
>
> <name>dfs.https.port</name>
> <value>50470</value>
>
> <name>dfs.https.address</name>
> <value>5.6.7.11:50470</value>
>
> <name>dfs.secondary.http.address</name>
> <value>5.6.7.12:50090</value>
> --------------------------------------
>
> server 3:
> core-site.xml
> --------------------------------------
> <name>fs.default.name</name>
> <value>hdfs://5.6.7.11:8020</value>
> --------------------------------------
>
> hdfs-site.xml
> --------------------------------------
> <name>dfs.datanode.address</name>
> <value>0.0.0.0:50010</value>
>
> <name>dfs.datanode.http.address</name>
> <value>0.0.0.0:50075</value>
>
> <name>dfs.http.address</name>
> <value>127.0.0.1:50070</value>
>
> <name>dfs.secondary.https.port</name>
> <value>50490</value>
>
> <name>dfs.https.port</name>
> <value>50470</value>
>
> <name>dfs.https.address</name>
> <value>127.0.0.1:50470</value>
>
> <name>dfs.secondary.http.address</name>
> <value>5.6.7.12:50090</value>
> --------------------------------------
>
> netstat output:
> server 1
>
> tcp        0      0 5.6.7.11:8020           0.0.0.0:*               LISTEN
> 10870/java
>
> tcp        0      0 5.6.7.11:50070          0.0.0.0:*               LISTEN