hi all,
I found hdfs du periodicity(one hour), and because my disk is big, the
smallest one is 15T, so when hdfs exec du, datanode will not respond for
about 3 minuts because of io loading, this cause a lot of problem, anybody
knows why hdfs doing this and how to disable it?

--
Thanks &  Regards
Xibin Liu

Hi,

HDFS does this to estimate space reports. Perhaps the discussion here
may help you: http://search-hadoop.com/m/LLBgUiH0Bg2

On Thu, Jan 24, 2013 at 12:51 PM, Xibin Liu <[EMAIL PROTECTED]> wrote:
> hi all,
> I found hdfs du periodicity(one hour), and because my disk is big, the
> smallest one is 15T, so when hdfs exec du, datanode will not respond for
> about 3 minuts because of io loading, this cause a lot of problem, anybody
> knows why hdfs doing this and how to disable it?
>
> --
> Thanks &  Regards
> Xibin Liu
>

--
Harsh J

Thanks, http://search-hadoop.com/m/LLBgUiH0Bg2 is my issue , but I still
dont't know how to solve this problem, 3 minutes not respond once an hour
 is a big problem for me, any clue for this?

2013/1/24 Harsh J <[EMAIL PROTECTED]>

> Hi,
>
> HDFS does this to estimate space reports. Perhaps the discussion here
> may help you: http://search-hadoop.com/m/LLBgUiH0Bg2
>
> On Thu, Jan 24, 2013 at 12:51 PM, Xibin Liu <[EMAIL PROTECTED]>
> wrote:
> > hi all,
> > I found hdfs du periodicity(one hour), and because my disk is big, the
> > smallest one is 15T, so when hdfs exec du, datanode will not respond for
> > about 3 minuts because of io loading, this cause a lot of problem,
> anybody
> > knows why hdfs doing this and how to disable it?
> >
> > --
> > Thanks &  Regards
> > Xibin Liu
> >
>
>
>
> --
> Harsh J
>

--
Thanks & Best Regards
Xibin Liu

I missed the periodicity part of your question. Unfortunately the "du"
refresh interval is hard-coded today, although the "df" interval is
configurable. Perhaps this is a bug - I filed
https://issues.apache.org/jira/browse/HADOOP-9241 to make it configurable.

Also, your problem reminded me of a similar issue my team and I faced
once before, and
http://allthingshadoop.com/2011/05/20/faster-datanodes-with-less-wait-io-using-df-instead-of-du/
helped us temporarily there. Perhaps it may help you as well, its a
good tip.

On Thu, Jan 24, 2013 at 1:24 PM, Xibin Liu <[EMAIL PROTECTED]> wrote:
> Thanks, http://search-hadoop.com/m/LLBgUiH0Bg2 is my issue , but I still
> dont't know how to solve this problem, 3 minutes not respond once an hour
> is a big problem for me, any clue for this?
>
>
> 2013/1/24 Harsh J <[EMAIL PROTECTED]>
>>
>> Hi,
>>
>> HDFS does this to estimate space reports. Perhaps the discussion here
>> may help you: http://search-hadoop.com/m/LLBgUiH0Bg2
>>
>> On Thu, Jan 24, 2013 at 12:51 PM, Xibin Liu <[EMAIL PROTECTED]>
>> wrote:
>> > hi all,
>> > I found hdfs du periodicity(one hour), and because my disk is big, the
>> > smallest one is 15T, so when hdfs exec du, datanode will not respond for
>> > about 3 minuts because of io loading, this cause a lot of problem,
>> > anybody
>> > knows why hdfs doing this and how to disable it?
>> >
>> > --
>> > Thanks &  Regards
>> > Xibin Liu
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> Thanks & Best Regards
> Xibin Liu
>

--
Harsh J

What type of FS are you using  under HDFS?  XFS, ext3, ext4?  The type and
configuration of the underlying FS will impact performance.
Most notably, ext3 has a lock-up effect when flushing disk cache.

On Thu, Jan 24, 2013 at 2:54 AM, Xibin Liu <[EMAIL PROTECTED]> wrote:

> Thanks, http://search-hadoop.com/m/LLBgUiH0Bg2 is my issue , but I still
> dont't know how to solve this problem, 3 minutes not respond once an hour
>  is a big problem for me, any clue for this?
>
>
> 2013/1/24 Harsh J <[EMAIL PROTECTED]>
>
>> Hi,
>>
>> HDFS does this to estimate space reports. Perhaps the discussion here
>> may help you: http://search-hadoop.com/m/LLBgUiH0Bg2
>>
>> On Thu, Jan 24, 2013 at 12:51 PM, Xibin Liu <[EMAIL PROTECTED]>
>> wrote:
>> > hi all,
>> > I found hdfs du periodicity(one hour), and because my disk is big, the
>> > smallest one is 15T, so when hdfs exec du, datanode will not respond for
>> > about 3 minuts because of io loading, this cause a lot of problem,
>> anybody
>> > knows why hdfs doing this and how to disable it?
>> >
>> > --
>> > Thanks &  Regards
>> > Xibin Liu
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> Thanks & Best Regards
> Xibin Liu
>
>