Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Can I use the cores of each CPU to be the datanodes instead of CPU?


+
谭军 2011-08-08, 08:46
+
Subroto Sanyal 2011-08-08, 09:05
+
谭军 2011-08-08, 09:19
Copy link to this message
-
RE: RE: Can I use the cores of each CPU to be the datanodes instead of CPU?
Hi Jun Tan,

 

   As Subroto pointed Hadoop doesn’t  depend on the number of cores but we
can tune the parameters based on the no of cores and type of job we are
running for better performance.

 

We can start multiple data nodes or task trackers irrespective of single
core or multi core by doing some changes in the script and configuration
files.

 

You can have a look into this, for what all changes need to do for starting
multiple data nodes in a single machine.

 

http://www.mail-archive.com/[EMAIL PROTECTED]/msg01353.html

Devaraj K

  _____  

From: 谭军 [mailto:[EMAIL PROTECTED]]
Sent: Monday, August 08, 2011 2:49 PM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re:RE: Can I use the cores of each CPU to be the datanodes instead
of CPU?

 

Hi Subroto,

I'm sorry for my poor English.

 

Are you thinking about CPU core to Hadoop process mapping?

Maybe this is the issue.

 

2 computers with 2 CPUs.

Each CPU has 2 cores.

Now I have 2 physical datanodes.

Can I get 4 physical datanodes?

I don't know wether I make my point clear?
 

--

Regards!

Jun Tan
在 2011-08-08 17:05:31,"Subroto Sanyal" <[EMAIL PROTECTED]> 写道:

 

Hi Jun,

 

Sorry, I didn’t get your question properly…..  :-(

 

Are you thinking in terms of Virtual Machines running on same physical
server?

Are you thinking about CPU core to Hadoop process mapping?

 

As per my understanding, Hadoop doesn’t have intelligence of different
cores (ideally application should be independent of hardware details)

 

It is possible to run multiple instance of DataNode in same physical server,
provide the configurations are different for each DataNode process.

 

Regards,
Subroto Sanyal

  _____  

From: 谭军 [mailto:[EMAIL PROTECTED]]
Sent: Monday, August 08, 2011 2:16 PM
To: mapreduce
Subject: Can I use the cores of each CPU to be the datanodes instead of CPU?

 

Hi,

I have 2 computers each of which is double-core CPU.

The 2 computers are datanodes and another computer is the namenode.

Now, I want to increase the number of datanode but no more computers
available.

Can I use the 4 cores of the 2 computers to be 4 datanodes?
--

Best regards!

Jun Tan


+
Subroto Sanyal 2011-08-08, 09:59
+
Arun C Murthy 2011-08-08, 17:42
+
Mayuresh 2011-08-09, 04:11
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB