Hadoop, mail # general - Re: dfs.data.dir

Re: dfs.data.dir

Eli Collins 2010-04-21, 21:33

Hey Mag,

You can bring down the datanode daemon, add the extra dfs.data.dir and
then restart. Since blocks are round robin'd the new directory will
have lower utilization (one other directories are full it will start
catching up). If that's not OK you can re-balance the directories by
hand with cp when the datanode is down (before you restart it). If
this takes you longer than 10 minutes the blocks on that datanode will
start getting re-replicated but when you bring the datanode back up
the namenode will notice the over-replicated blocks and remove them.

Thanks,
Eli

On Wed, Apr 21, 2010 at 4:09 AM, Mag Gam <[EMAIL PROTECTED]> wrote:
> I would like to add/remove more data directories to my hdfs installation.
>
> Currently, what I do is decommision the entire node and then remove
> all content from dfs.data.dir and renable the node. But is there an
> easier way? Each of my node consists of 2TB of data and I don't want
> to waste the time...
>

Re: dfs.data.dir

Steve Loughran 2010-04-22, 12:41

Eli Collins wrote:
> Hey Mag,
>
> You can bring down the datanode daemon, add the extra dfs.data.dir and
> then restart. Since blocks are round robin'd the new directory will
> have lower utilization (one other directories are full it will start
> catching up). If that's not OK you can re-balance the directories by
> hand with cp when the datanode is down (before you restart it). If
> this takes you longer than 10 minutes the blocks on that datanode will
> start getting re-replicated but when you bring the datanode back up
> the namenode will notice the over-replicated blocks and remove them.

that brings up a couple of issues I've been thinking about now that
workers can go to 6+ HDDs/node

* a way to measure the distribution across disks, rather than just
nodes. DfsClient doesn't provide enough info here yet.
* a way to triger some rebalancing on a single node, to say "position
stuff more fairly". You don't need to worry about network traffic, just
local disk load and CPU time, so it should be simpler.

Re: dfs.data.dir

Allen Wittenauer 2010-04-22, 19:59

On Apr 22, 2010, at 5:41 AM, Steve Loughran wrote:
> that brings up a couple of issues I've been thinking about now that workers can go to 6+ HDDs/node
>
> * a way to measure the distribution across disks, rather than just nodes. DfsClient doesn't provide enough info here yet.

What should probably happen is that instead of throwing you to the file browser, clicking on a host from the live nodes page should probably put you on a "stats about this node" page.

> * a way to triger some rebalancing on a single node, to say "position stuff more fairly". You don't need to worry about network traffic, just local disk load and CPU time, so it should be simpler.
Yup. Working with 8 drives per node, it is interesting to see how unbalanced the data gets after a while. [Luckily, we have MR tmp space segregated off so I'm sure it would be a lot worse if we didn't!]

Someone should file a jira. :)

Re: dfs.data.dir

Eli Collins 2010-04-23, 02:35

On Thu, Apr 22, 2010 at 12:59 PM, Allen Wittenauer
<[EMAIL PROTECTED]> wrote:
>
>> * a way to triger some rebalancing on a single node, to say "position stuff more fairly". You don't need to worry about network traffic, just local disk load and CPU time, so it should be simpler.
>
>
> Yup. Working with 8 drives per node, it is interesting to see how unbalanced the data gets after a while. [Luckily, we have MR tmp space segregated off so I'm sure it would be a lot worse if we didn't!]

How unbalanced do you see your mounts? I've mostly heard of this
being an issue when new disks are added.

Thanks,
Eli

Re: dfs.data.dir

Steve Loughran 2010-04-26, 16:45

Allen Wittenauer wrote:
> On Apr 22, 2010, at 5:41 AM, Steve Loughran wrote:
>> that brings up a couple of issues I've been thinking about now that workers can go to 6+ HDDs/node
>>
>> * a way to measure the distribution across disks, rather than just nodes. DfsClient doesn't provide enough info here yet.
>
> What should probably happen is that instead of throwing you to the file browser, clicking on a host from the live nodes page should probably put you on a "stats about this node" page.

I don't want to do any of this by hand. I want machine readable content
something can aggregate over time.

>
>> * a way to triger some rebalancing on a single node, to say "position stuff more fairly". You don't need to worry about network traffic, just local disk load and CPU time, so it should be simpler.
>
>
> Yup. Working with 8 drives per node, it is interesting to see how unbalanced the data gets after a while. [Luckily, we have MR tmp space segregated off so I'm sure it would be a lot worse if we didn't!]
>
> Someone should file a jira. :)

Especially if someone else offers to fix it.

Re: dfs.data.dir

Scott Carey 2010-04-28, 17:31

On Apr 26, 2010, at 9:45 AM, Steve Loughran wrote:

> Allen Wittenauer wrote:
>> On Apr 22, 2010, at 5:41 AM, Steve Loughran wrote:
>>> that brings up a couple of issues I've been thinking about now that workers can go to 6+ HDDs/node
>>>
>>> * a way to measure the distribution across disks, rather than just nodes. DfsClient doesn't provide enough info here yet.
>>
>> What should probably happen is that instead of throwing you to the file browser, clicking on a host from the live nodes page should probably put you on a "stats about this node" page.
>
> I don't want to do any of this by hand. I want machine readable content
> something can aggregate over time.
>
>>
>>> * a way to triger some rebalancing on a single node, to say "position stuff more fairly". You don't need to worry about network traffic, just local disk load and CPU time, so it should be simpler.
>>
>>
>> Yup. Working with 8 drives per node, it is interesting to see how unbalanced the data gets after a while. [Luckily, we have MR tmp space segregated off so I'm sure it would be a lot worse if we didn't!]
>>
>> Someone should file a jira. :)
>
> Especially if someone else offers to fix it.
>

Should be trivial to at least make the new block allocation choose which device to allocate the block on with a weighted roulette algorithm instead of round-robin.

Re: dfs.data.dir

Steve Loughran 2010-04-29, 09:28

Scott Carey wrote:
> On Apr 26, 2010, at 9:45 AM, Steve Loughran wrote:
>
>> Allen Wittenauer wrote:
>>> On Apr 22, 2010, at 5:41 AM, Steve Loughran wrote:
>>>> that brings up a couple of issues I've been thinking about now that workers can go to 6+ HDDs/node
>>>>
>>>> * a way to measure the distribution across disks, rather than just nodes. DfsClient doesn't provide enough info here yet.
>>> What should probably happen is that instead of throwing you to the file browser, clicking on a host from the live nodes page should probably put you on a "stats about this node" page.
>> I don't want to do any of this by hand. I want machine readable content
>> something can aggregate over time.
>>
>>>> * a way to triger some rebalancing on a single node, to say "position stuff more fairly". You don't need to worry about network traffic, just local disk load and CPU time, so it should be simpler.
>>>
>>> Yup. Working with 8 drives per node, it is interesting to see how unbalanced the data gets after a while. [Luckily, we have MR tmp space segregated off so I'm sure it would be a lot worse if we didn't!]
>>>
>>> Someone should file a jira. :)
>> Especially if someone else offers to fix it.
>>
>
> Should be trivial to at least make the new block allocation choose which device to allocate the block on with a weighted roulette algorithm instead of round-robin.
>

I'd go for another plugin point with the default impl being round-robin,
let people come up with other strategies, and add a way from DfsClient
to measure distribution.

Other strategies could use specific knowledge about the disks -are they
raided, can you hot swap, is the disk playing up, etc, etc, feature
creep that I encourage people to explore.