Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> DFS respond very slow


+
Alexey 2012-10-09, 10:12
+
Alexey 2012-10-10, 05:23
+
Harsh J 2012-10-10, 06:50
+
Alexey 2012-10-10, 06:54
+
Harsh J 2012-10-10, 06:56
+
Alexey 2012-10-10, 07:20
+
Vinod Kumar Vavilapalli 2012-10-16, 00:22
Copy link to this message
-
Re: DFS respond very slow

I just realized one more thing. You mentioned "disk is 700Gb RAID".  How many disks overall? What RAID configuration? Usually we advocate JBOD with hadoop to avoid performance hits with RAID, and let HDFS itself take care of replication. May be you are running into this?

Thanks,
+Vinod

On Oct 15, 2012, at 5:22 PM, Vinod Kumar Vavilapalli wrote:

> Try picking up a single operation say "hadoop dfs -ls" and start profiling.
>  - Time the client JVM is taking to start. Enable debug logging on the client side by exporting HADOOP_ROOT_LOGGER=DEBUG,CONSOLE
>  - Time between the client starting and the namenode audit logs showing the read request. Also enable debug logging on the daemons too.
>  - Also, you can wget the namenode web pages and see how fast they return.
>
> To repeat what is already obvious, It is most likely related to your network setup and/or configuration.
>
> Thanks,
> +Vinod
>
> On Oct 10, 2012, at 12:20 AM, Alexey wrote:
>
>> ok, here you go:
>> I have 3 servers:
>> datanode on server 1, 2, 3
>> namenode on server 1
>> secondarynamenode on server 2
>>
>> all servers are at the hetzner datacenter and connected through 100Mbit
>> link, pings between them about 0.1ms
>>
>> each server has 24Gb ram and intel core i7 3Ghz CPU
>> disk is 700Gb RAID
>>
>> the bindings related configuration is the following:
>> server 1:
>> core-site.xml
>> --------------------------------------
>> <name>fs.default.name</name>
>> <value>hdfs://5.6.7.11:8020</value>
>> --------------------------------------
>>
>> hdfs-site.xml
>> --------------------------------------
>> <name>dfs.datanode.address</name>
>> <value>0.0.0.0:50010</value>
>>
>> <name>dfs.datanode.http.address</name>
>> <value>0.0.0.0:50075</value>
>>
>> <name>dfs.http.address</name>
>> <value>5.6.7.11:50070</value>
>>
>> <name>dfs.secondary.https.port</name>
>> <value>50490</value>
>>
>> <name>dfs.https.port</name>
>> <value>50470</value>
>>
>> <name>dfs.https.address</name>
>> <value>5.6.7.11:50470</value>
>>
>> <name>dfs.secondary.http.address</name>
>> <value>5.6.7.12:50090</value>
>> --------------------------------------
>>
>> server 2:
>> core-site.xml
>> --------------------------------------
>> <name>fs.default.name</name>
>> <value>hdfs://5.6.7.11:8020</value>
>> --------------------------------------
>>
>> hdfs-site.xml
>> --------------------------------------
>> <name>dfs.datanode.address</name>
>> <value>0.0.0.0:50010</value>
>>
>> <name>dfs.datanode.http.address</name>
>> <value>0.0.0.0:50075</value>
>>
>> <name>dfs.http.address</name>
>> <value>5.6.7.11:50070</value>
>>
>> <name>dfs.secondary.https.port</name>
>> <value>50490</value>
>>
>> <name>dfs.https.port</name>
>> <value>50470</value>
>>
>> <name>dfs.https.address</name>
>> <value>5.6.7.11:50470</value>
>>
>> <name>dfs.secondary.http.address</name>
>> <value>5.6.7.12:50090</value>
>> --------------------------------------
>>
>> server 3:
>> core-site.xml
>> --------------------------------------
>> <name>fs.default.name</name>
>> <value>hdfs://5.6.7.11:8020</value>
>> --------------------------------------
>>
>> hdfs-site.xml
>> --------------------------------------
>> <name>dfs.datanode.address</name>
>> <value>0.0.0.0:50010</value>
>>
>> <name>dfs.datanode.http.address</name>
>> <value>0.0.0.0:50075</value>
>>
>> <name>dfs.http.address</name>
>> <value>127.0.0.1:50070</value>
>>
>> <name>dfs.secondary.https.port</name>
>> <value>50490</value>
>>
>> <name>dfs.https.port</name>
>> <value>50470</value>
>>
>> <name>dfs.https.address</name>
>> <value>127.0.0.1:50470</value>
>>
>> <name>dfs.secondary.http.address</name>
>> <value>5.6.7.12:50090</value>
>> --------------------------------------
>>
>> netstat output:
>> server 1
>>> tcp        0      0 5.6.7.11:8020           0.0.0.0:*               LISTEN      10870/java
>>> tcp        0      0 5.6.7.11:50070          0.0.0.0:*               LISTEN      10870/java
>>> tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      10997/java
+
Andy Isaacson 2012-10-16, 01:56
+
Ted Dunning 2012-10-16, 02:23
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB