Hi all,

Is it possible to run multiple datanode instances on single node hadoop
cluster?

How to do that?

thanks,
-Ajit

It is possible to run multiple datanodes.
You would have to have separate conf file for each datanode, basically you need to have different datanode address (dfs.datanode.address and related)
and different storage directories dfs.data.dir. Have different configs and start each datanode instance with different conf directory containing different conf file
Lohit

----- Original Message ----
From: Ajit Ratnaparkhi <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wednesday, February 25, 2009 11:04:15 PM
Subject: multiple datanodes on single node cluster

Hi all,

Is it possible to run multiple datanode instances on single node hadoop
cluster?

How to do that?

thanks,
-Ajit

Hey,

I'd add it's possible, but not necessary for day-to-day operations.

We've got one data node instance which manages 48 disks... the single  
node can probably support all the disk activity you have.

I'm not sure how certain activities, such as putting a node into the  
hostsExclude, would work with multiple data nodes.  One way to find out!

Brian

On Feb 26, 2009, at 1:55 AM, lohit wrote:

> It is possible to run multiple datanodes.
> You would have to have separate conf file for each datanode,  
> basically you need to have different datanode address  
> (dfs.datanode.address and related)
> and different storage directories dfs.data.dir. Have different  
> configs and start each datanode instance with different conf  
> directory containing different conf file
> Lohit
>
>
>
> ----- Original Message ----
> From: Ajit Ratnaparkhi <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wednesday, February 25, 2009 11:04:15 PM
> Subject: multiple datanodes on single node cluster
>
> Hi all,
>
> Is it possible to run multiple datanode instances on single node  
> hadoop
> cluster?
>
> How to do that?
>
> thanks,
> -Ajit

I think the advantage might be due to current locking structure that can
pause the datanode during the block report or during large deletes.

Once the lock granularity is finer and there is less chance of a long
operation blocking the datanode, this will be less interesting.

On Thu, Feb 26, 2009 at 5:04 AM, Brian Bockelman <[EMAIL PROTECTED]>wrote:

> Hey,
>
> I'd add it's possible, but not necessary for day-to-day operations.
>
> We've got one data node instance which manages 48 disks... the single node
> can probably support all the disk activity you have.
>
> I'm not sure how certain activities, such as putting a node into the
> hostsExclude, would work with multiple data nodes.  One way to find out!
>
> Brian
>
>
> On Feb 26, 2009, at 1:55 AM, lohit wrote:
>
>  It is possible to run multiple datanodes.
>> You would have to have separate conf file for each datanode, basically you
>> need to have different datanode address (dfs.datanode.address and related)
>> and different storage directories dfs.data.dir. Have different configs and
>> start each datanode instance with different conf directory containing
>> different conf file
>> Lohit
>>
>>
>>
>> ----- Original Message ----
>> From: Ajit Ratnaparkhi <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Sent: Wednesday, February 25, 2009 11:04:15 PM
>> Subject: multiple datanodes on single node cluster
>>
>> Hi all,
>>
>> Is it possible to run multiple datanode instances on single node hadoop
>> cluster?
>>
>> How to do that?
>>
>> thanks,
>> -Ajit
>>
>
>

On Feb 25, 2009, at 11:55 PM, lohit wrote:

> It is possible to run multiple datanodes.
> You would have to have separate conf file for each datanode,  
> basically you need to have different datanode address  
> (dfs.datanode.address and related)
> and different storage directories dfs.data.dir. Have different  
> configs and start each datanode instance with different conf  
> directory containing different conf file
> Lohit
>
>
>
> ----- Original Message ----
> From: Ajit Ratnaparkhi <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wednesday, February 25, 2009 11:04:15 PM
> Subject: multiple datanodes on single node cluster
>
> Hi all,
>
> Is it possible to run multiple datanode instances on single node  
> hadoop
> cluster?
>
> How to do that?
>
> thanks,
> -Ajit
>
>
The minicluster also allows multiple data nodes in the same JVM.
This normally used only for testing. See DataNodeCluster under src/test.
This is probably not what you are looking for but I thought I would  
mention it.
sanjay