Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> DFS respond very slow


+
Alexey 2012-10-09, 10:12
+
Alexey 2012-10-10, 05:23
+
Harsh J 2012-10-10, 06:50
+
Alexey 2012-10-10, 06:54
+
Harsh J 2012-10-10, 06:56
+
Alexey 2012-10-10, 07:20
Copy link to this message
-
Re: DFS respond very slow
Try picking up a single operation say "hadoop dfs -ls" and start profiling.
 - Time the client JVM is taking to start. Enable debug logging on the client side by exporting HADOOP_ROOT_LOGGER=DEBUG,CONSOLE
 - Time between the client starting and the namenode audit logs showing the read request. Also enable debug logging on the daemons too.
 - Also, you can wget the namenode web pages and see how fast they return.

To repeat what is already obvious, It is most likely related to your network setup and/or configuration.

Thanks,
+Vinod

On Oct 10, 2012, at 12:20 AM, Alexey wrote:

> ok, here you go:
> I have 3 servers:
> datanode on server 1, 2, 3
> namenode on server 1
> secondarynamenode on server 2
>
> all servers are at the hetzner datacenter and connected through 100Mbit
> link, pings between them about 0.1ms
>
> each server has 24Gb ram and intel core i7 3Ghz CPU
> disk is 700Gb RAID
>
> the bindings related configuration is the following:
> server 1:
> core-site.xml
> --------------------------------------
> <name>fs.default.name</name>
> <value>hdfs://5.6.7.11:8020</value>
> --------------------------------------
>
> hdfs-site.xml
> --------------------------------------
> <name>dfs.datanode.address</name>
> <value>0.0.0.0:50010</value>
>
> <name>dfs.datanode.http.address</name>
> <value>0.0.0.0:50075</value>
>
> <name>dfs.http.address</name>
> <value>5.6.7.11:50070</value>
>
> <name>dfs.secondary.https.port</name>
> <value>50490</value>
>
> <name>dfs.https.port</name>
> <value>50470</value>
>
> <name>dfs.https.address</name>
> <value>5.6.7.11:50470</value>
>
> <name>dfs.secondary.http.address</name>
> <value>5.6.7.12:50090</value>
> --------------------------------------
>
> server 2:
> core-site.xml
> --------------------------------------
> <name>fs.default.name</name>
> <value>hdfs://5.6.7.11:8020</value>
> --------------------------------------
>
> hdfs-site.xml
> --------------------------------------
> <name>dfs.datanode.address</name>
> <value>0.0.0.0:50010</value>
>
> <name>dfs.datanode.http.address</name>
> <value>0.0.0.0:50075</value>
>
> <name>dfs.http.address</name>
> <value>5.6.7.11:50070</value>
>
> <name>dfs.secondary.https.port</name>
> <value>50490</value>
>
> <name>dfs.https.port</name>
> <value>50470</value>
>
> <name>dfs.https.address</name>
> <value>5.6.7.11:50470</value>
>
> <name>dfs.secondary.http.address</name>
> <value>5.6.7.12:50090</value>
> --------------------------------------
>
> server 3:
> core-site.xml
> --------------------------------------
> <name>fs.default.name</name>
> <value>hdfs://5.6.7.11:8020</value>
> --------------------------------------
>
> hdfs-site.xml
> --------------------------------------
> <name>dfs.datanode.address</name>
> <value>0.0.0.0:50010</value>
>
> <name>dfs.datanode.http.address</name>
> <value>0.0.0.0:50075</value>
>
> <name>dfs.http.address</name>
> <value>127.0.0.1:50070</value>
>
> <name>dfs.secondary.https.port</name>
> <value>50490</value>
>
> <name>dfs.https.port</name>
> <value>50470</value>
>
> <name>dfs.https.address</name>
> <value>127.0.0.1:50470</value>
>
> <name>dfs.secondary.http.address</name>
> <value>5.6.7.12:50090</value>
> --------------------------------------
>
> netstat output:
> server 1
>> tcp        0      0 5.6.7.11:8020           0.0.0.0:*               LISTEN      10870/java
>> tcp        0      0 5.6.7.11:50070          0.0.0.0:*               LISTEN      10870/java
>> tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      10997/java
>> tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      10997/java
>> tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      10997/java
>
> server 2
>> tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      23683/java
>> tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      23683/java
>> tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      23683/java
+
Vinod Kumar Vavilapalli 2012-10-16, 02:36
+
Andy Isaacson 2012-10-16, 01:56
+
Ted Dunning 2012-10-16, 02:23