Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> multiple nodes one machine


Copy link to this message
-
Re: multiple nodes one machine
Hi,

With my configuration in place, I simply do:

"hadoop datanode 2>&1 > /tmp/datanode-$RANDOM.log &" required number
of times. I then track the launched instances with "jps", so I can
send them quit signals when I want to tear them down again.

On Tue, Apr 10, 2012 at 1:52 AM, Barry, Sean F <[EMAIL PROTECTED]> wrote:
> Harsh,
> I am interested in adding datanodes just for testing.
>
>
> I have a few more things I should have said earlier.
> My current cluster looks like this. Which I set up exactly like tutorial http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/ except I am running Suse 12.1 on both boxes.
>
> Master - running NameNode, DataNode, TaskTracker, SecNameNode and JobTtracker
> ------
> Slave - running Datanode and TaskTracker
>
> My (4 core) slave  machine is the one that I would like to add three additional datanodes to But when I use the run-additionalDN.sh script I get an Usage: java DataNode    [-rollback].
>
> Am I supposed to run the script on my master node or slave node?
>
> -SB
>
>
>
> -----Original Message-----
> From: Harsh J [mailto:[EMAIL PROTECTED]]
> Sent: Monday, April 09, 2012 11:35 AM
> To: [EMAIL PROTECTED]
> Subject: Re: multiple nodes one machine
>
> Barry,
>
> Depends on what you'll be testing. If you want more daemons, then yes you need to add more nodes onto the same box (configs may be tweaked to achieve this). If you just want MR to provide more slots for tasks, then a specific task tracker property alone may be edited.
>
> For more daemons, see http://search-hadoop.com/m/a4klk28NUr12 and a neat config I use for running them without too much config mess:
> https://gist.github.com/2345300
>
> For the latter, see:
> http://wiki.apache.org/hadoop/FAQ#I_see_a_maximum_of_2_maps.2BAC8-reduces_spawned_concurrently_on_each_TaskTracker.2C_how_do_I_increase_that.3F
>
> Alternatively, use the hadoop-test jar provided classes:
> MiniDFSCluster and MiniMRCluster which can run from with a test suite itself (With multiple threads as daemons, to simply test around with).
>
> On Mon, Apr 9, 2012 at 9:52 PM, Barry, Sean F <[EMAIL PROTECTED]> wrote:
>> Hi all,
>>
>> I currently have a 2 node cluster up and running. But now I face a new issue, one of my nodes is running a Datanode and a Tasktracker on a 4 core machine and in order to do a bit of proof of concept testing I would like to have 4 nodes running on that particular machine. Does this mean that I would need to set that up as a pseudodistributed cluster? or do you have any other suggestions? And would I need to add 3 more datanodes and 3 more tasktrackers or either or?
>>
>> Thanks
>> -SB
>
>
>
> --
> Harsh J

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB