hi all,
I found hdfs du periodicity(one hour), and because my disk is big, the
smallest one is 15T, so when hdfs exec du, datanode will not respond for
about 3 minuts because of io loading, this cause a lot of problem, anybody
knows why hdfs doing this and how to disable it?

--
Thanks &  Regards
Xibin Liu

Hi,

HDFS does this to estimate space reports. Perhaps the discussion here
may help you: http://search-hadoop.com/m/LLBgUiH0Bg2

On Thu, Jan 24, 2013 at 12:51 PM, Xibin Liu <[EMAIL PROTECTED]> wrote:
> hi all,
> I found hdfs du periodicity(one hour), and because my disk is big, the
> smallest one is 15T, so when hdfs exec du, datanode will not respond for
> about 3 minuts because of io loading, this cause a lot of problem, anybody
> knows why hdfs doing this and how to disable it?
>
> --
> Thanks &  Regards
> Xibin Liu
>

--
Harsh J

Thanks, http://search-hadoop.com/m/LLBgUiH0Bg2 is my issue , but I still
dont't know how to solve this problem, 3 minutes not respond once an hour
 is a big problem for me, any clue for this?

2013/1/24 Harsh J <[EMAIL PROTECTED]>

> Hi,
>
> HDFS does this to estimate space reports. Perhaps the discussion here
> may help you: http://search-hadoop.com/m/LLBgUiH0Bg2
>
> On Thu, Jan 24, 2013 at 12:51 PM, Xibin Liu <[EMAIL PROTECTED]>
> wrote:
> > hi all,
> > I found hdfs du periodicity(one hour), and because my disk is big, the
> > smallest one is 15T, so when hdfs exec du, datanode will not respond for
> > about 3 minuts because of io loading, this cause a lot of problem,
> anybody
> > knows why hdfs doing this and how to disable it?
> >
> > --
> > Thanks &  Regards
> > Xibin Liu
> >
>
>
>
> --
> Harsh J
>

--
Thanks & Best Regards
Xibin Liu

What type of FS are you using  under HDFS?  XFS, ext3, ext4?  The type and
configuration of the underlying FS will impact performance.
Most notably, ext3 has a lock-up effect when flushing disk cache.

On Thu, Jan 24, 2013 at 2:54 AM, Xibin Liu <[EMAIL PROTECTED]> wrote:

> Thanks, http://search-hadoop.com/m/LLBgUiH0Bg2 is my issue , but I still
> dont't know how to solve this problem, 3 minutes not respond once an hour
>  is a big problem for me, any clue for this?
>
>
> 2013/1/24 Harsh J <[EMAIL PROTECTED]>
>
>> Hi,
>>
>> HDFS does this to estimate space reports. Perhaps the discussion here
>> may help you: http://search-hadoop.com/m/LLBgUiH0Bg2
>>
>> On Thu, Jan 24, 2013 at 12:51 PM, Xibin Liu <[EMAIL PROTECTED]>
>> wrote:
>> > hi all,
>> > I found hdfs du periodicity(one hour), and because my disk is big, the
>> > smallest one is 15T, so when hdfs exec du, datanode will not respond for
>> > about 3 minuts because of io loading, this cause a lot of problem,
>> anybody
>> > knows why hdfs doing this and how to disable it?
>> >
>> > --
>> > Thanks &  Regards
>> > Xibin Liu
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> Thanks & Best Regards
> Xibin Liu
>
>

I missed the periodicity part of your question. Unfortunately the "du"
refresh interval is hard-coded today, although the "df" interval is
configurable. Perhaps this is a bug - I filed
https://issues.apache.org/jira/browse/HADOOP-9241 to make it configurable.

Also, your problem reminded me of a similar issue my team and I faced
once before, and
http://allthingshadoop.com/2011/05/20/faster-datanodes-with-less-wait-io-using-df-instead-of-du/
helped us temporarily there. Perhaps it may help you as well, its a
good tip.

On Thu, Jan 24, 2013 at 1:24 PM, Xibin Liu <[EMAIL PROTECTED]> wrote:
> Thanks, http://search-hadoop.com/m/LLBgUiH0Bg2 is my issue , but I still
> dont't know how to solve this problem, 3 minutes not respond once an hour
> is a big problem for me, any clue for this?
>
>
> 2013/1/24 Harsh J <[EMAIL PROTECTED]>
>>
>> Hi,
>>
>> HDFS does this to estimate space reports. Perhaps the discussion here
>> may help you: http://search-hadoop.com/m/LLBgUiH0Bg2
>>
>> On Thu, Jan 24, 2013 at 12:51 PM, Xibin Liu <[EMAIL PROTECTED]>
>> wrote:
>> > hi all,
>> > I found hdfs du periodicity(one hour), and because my disk is big, the
>> > smallest one is 15T, so when hdfs exec du, datanode will not respond for
>> > about 3 minuts because of io loading, this cause a lot of problem,
>> > anybody
>> > knows why hdfs doing this and how to disable it?
>> >
>> > --
>> > Thanks &  Regards
>> > Xibin Liu
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> Thanks & Best Regards
> Xibin Liu
>

--
Harsh J