Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Re: DFS respond very slow


Copy link to this message
-
Re: DFS respond very slow
Alexey 2012-10-15, 16:49
Hi again,
is anybody can help me? I'm ready to send you a reasonable donation for
the help.

On 10/09/12 11:56 PM, Harsh J wrote:
> Hi,
>
> OK, can you detail your network infrastructure used here, and also
> make sure your daemons are binding to the right interfaces as well
> (use netstat to check perhaps)? What rate of transfer do you get for
> simple file transfers (ftp, scp, etc.)?
>
> On Wed, Oct 10, 2012 at 12:24 PM, Alexey <[EMAIL PROTECTED]> wrote:
>> Hello Harsh,
>>
>> I notices such issues from the start.
>> Yes, I mean dfs.balance.bandwidthPerSec property, I set this property to
>> 5000000.
>>
>> On 10/09/12 11:50 PM, Harsh J wrote:
>>> Hey Alexey,
>>>
>>> Have you noticed this right from the start itself? Also, what exactly
>>> do you mean by "Limited replication bandwidth between datanodes -
>>> 5Mb." - Are you talking of dfs.balance.bandwidthPerSec property?
>>>
>>> On Wed, Oct 10, 2012 at 10:53 AM, Alexey <[EMAIL PROTECTED]> wrote:
>>>> Additional info: I also tried to use openjdk instead of sun's - issue
>>>> still persists
>>>>
>>>> On 10/09/12 03:12 AM, Alexey wrote:
>>>>> Hi,
>>>>>
>>>>> I have an issues with hadoop dfs, I have 3 servers (24Gb RAM on each).
>>>>> The servers are not overloaded, they just have hadoop installed. One
>>>>> have datanode and namenode, second - datanode only, third - datanode and
>>>>> secondarynamenode.
>>>>>
>>>>> Hadoop datanodes have a max memory limit 8Gb. Default replication factor
>>>>> - 2. Limited replication bandwidth between datanodes - 5Mb.
>>>>>
>>>>> I've setupped hadoop to communicate between nodes by IP address.
>>>>> Everything is works - I can read/write files on each datanode, etc. But
>>>>> the issue is that hadoop dfs commands are executing very slow, even
>>>>> "hadoop dfs -ls /" takes about 3 seconds to execute, but it have only
>>>>> one folder /user in it.
>>>>> Files are also uploading to the hdfs very slow - hundreds kilobytes/second.
>>>>>
>>>>> I'm using Debian stable x86-64 distribution and hadoop running through
>>>>> sun-java6-jdk 6.26-0squeeze1
>>>>>
>>>>> Please give me any suggestions what I need to adjust/check to arrange
>>>>> this issue.
>>>>>
>>>>> As I said before - overall hdfs configuration is correct, because
>>>>> everything works except performance.
>>>>>
>>>>> --
>>>>> Best regards
>>>>> Alexey
>>>>>
>>>>
>>>> --
>>>> Best regards
>>>> Alexey
>>>
>>>
>>>
>>
>> --
>> Best regards
>> Alexey
>
>
>

--
Best regards
Alexey