|
|
-
Re: DFS respond very slowAlexey 2012-10-15, 16:49
Hi again,
is anybody can help me? I'm ready to send you a reasonable donation for the help. On 10/09/12 11:56 PM, Harsh J wrote: > Hi, > > OK, can you detail your network infrastructure used here, and also > make sure your daemons are binding to the right interfaces as well > (use netstat to check perhaps)? What rate of transfer do you get for > simple file transfers (ftp, scp, etc.)? > > On Wed, Oct 10, 2012 at 12:24 PM, Alexey <[EMAIL PROTECTED]> wrote: >> Hello Harsh, >> >> I notices such issues from the start. >> Yes, I mean dfs.balance.bandwidthPerSec property, I set this property to >> 5000000. >> >> On 10/09/12 11:50 PM, Harsh J wrote: >>> Hey Alexey, >>> >>> Have you noticed this right from the start itself? Also, what exactly >>> do you mean by "Limited replication bandwidth between datanodes - >>> 5Mb." - Are you talking of dfs.balance.bandwidthPerSec property? >>> >>> On Wed, Oct 10, 2012 at 10:53 AM, Alexey <[EMAIL PROTECTED]> wrote: >>>> Additional info: I also tried to use openjdk instead of sun's - issue >>>> still persists >>>> >>>> On 10/09/12 03:12 AM, Alexey wrote: >>>>> Hi, >>>>> >>>>> I have an issues with hadoop dfs, I have 3 servers (24Gb RAM on each). >>>>> The servers are not overloaded, they just have hadoop installed. One >>>>> have datanode and namenode, second - datanode only, third - datanode and >>>>> secondarynamenode. >>>>> >>>>> Hadoop datanodes have a max memory limit 8Gb. Default replication factor >>>>> - 2. Limited replication bandwidth between datanodes - 5Mb. >>>>> >>>>> I've setupped hadoop to communicate between nodes by IP address. >>>>> Everything is works - I can read/write files on each datanode, etc. But >>>>> the issue is that hadoop dfs commands are executing very slow, even >>>>> "hadoop dfs -ls /" takes about 3 seconds to execute, but it have only >>>>> one folder /user in it. >>>>> Files are also uploading to the hdfs very slow - hundreds kilobytes/second. >>>>> >>>>> I'm using Debian stable x86-64 distribution and hadoop running through >>>>> sun-java6-jdk 6.26-0squeeze1 >>>>> >>>>> Please give me any suggestions what I need to adjust/check to arrange >>>>> this issue. >>>>> >>>>> As I said before - overall hdfs configuration is correct, because >>>>> everything works except performance. >>>>> >>>>> -- >>>>> Best regards >>>>> Alexey >>>>> >>>> >>>> -- >>>> Best regards >>>> Alexey >>> >>> >>> >> >> -- >> Best regards >> Alexey > > > -- Best regards Alexey |