Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Can I use the cores of each CPU to be the datanodes instead of CPU?


+
谭军 2011-08-08, 08:46
+
Subroto Sanyal 2011-08-08, 09:05
+
谭军 2011-08-08, 09:19
+
Devaraj K 2011-08-08, 09:37
Copy link to this message
-
RE: RE: Can I use the cores of each CPU to be the datanodes instead of CPU?
Hi Jun,

 

User can run multiple DataNode instances on same physical server provided
user changes the DataNode configuration.

 

One simple test you can try:

Copy the DataNode installation to a different folder/directory of same
physical server.

Change the necessary configuration like dfs.data.dir/http ports etc.

Start both the DataNodes. It should run properly.

 

We have tried this for development and testing purpose but, real case this
is usually not the scenario.

 

Please visit the following link:

http://hadoop.apache.org/common/docs/current/hdfs_design.html

The link clearly tells:

“The architecture does not preclude running multiple DataNodes on the same
machine”

 

Hope it answers your query. :-)

 

Regards,

Computers make very fast and accurate mistakes.......
This email and its attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above.
Any use of the information contained herein in any way (including, but not
limited to, total or partial disclosure, reproduction, or dissemination) by
persons other than the intended recipient(s) is prohibited.
If you receive this e-mail in error, please notify the sender by phone or
email immediately and delete it!

  _____  

From: 谭军 [mailto:[EMAIL PROTECTED]]
Sent: Monday, August 08, 2011 2:49 PM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re:RE: Can I use the cores of each CPU to be the datanodes instead
of CPU?

 

Hi Subroto,

I'm sorry for my poor English.

 

Are you thinking about CPU core to Hadoop process mapping?

Maybe this is the issue.

 

2 computers with 2 CPUs.

Each CPU has 2 cores.

Now I have 2 physical datanodes.

Can I get 4 physical datanodes?

I don't know wether I make my point clear?
 

--

Regards!

Jun Tan
在 2011-08-08 17:05:31,"Subroto Sanyal" <[EMAIL PROTECTED]> 写道:

 

Hi Jun,

 

Sorry, I didn’t get your question properly…..  :-(

 

Are you thinking in terms of Virtual Machines running on same physical
server?

Are you thinking about CPU core to Hadoop process mapping?

 

As per my understanding, Hadoop doesn’t have intelligence of different
cores (ideally application should be independent of hardware details)

 

It is possible to run multiple instance of DataNode in same physical server,
provide the configurations are different for each DataNode process.

 

Regards,
Subroto Sanyal

  _____  

From: 谭军 [mailto:[EMAIL PROTECTED]]
Sent: Monday, August 08, 2011 2:16 PM
To: mapreduce
Subject: Can I use the cores of each CPU to be the datanodes instead of CPU?

 

Hi,

I have 2 computers each of which is double-core CPU.

The 2 computers are datanodes and another computer is the namenode.

Now, I want to increase the number of datanode but no more computers
available.

Can I use the 4 cores of the 2 computers to be 4 datanodes?
--

Best regards!

Jun Tan


+
Arun C Murthy 2011-08-08, 17:42
+
Mayuresh 2011-08-09, 04:11