Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> dynamically resizing Hadoop cluster on AWS?


Copy link to this message
-
Re: dynamically resizing Hadoop cluster on AWS?
If only compute node (TaskTracker or NodeManager) in your instance, then decommission nodes and shutdown related EC2 instances should be fine although some finished/running tasks might need to be re-run automatically. If future, we would support gracefully decommission (tracked by YARN-914 and MAPREDUCE-5381) so that no tasks need to be rerun in this case (but need to wait a while).

Thanks,

Junping

----- Original Message -----
From: "Nan Zhu" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wednesday, October 23, 2013 8:15:51 PM
Subject: Re: dynamically resizing Hadoop cluster on AWS?

Oh, I’m not running HDFS in the instances, I use S3 to save data

--  
Nan Zhu
School of Computer Science,
McGill University

On Wednesday, October 23, 2013 at 11:11 PM, Nan Zhu wrote:

> Hi, all  
>  
> I’m running a Hadoop cluster on AWS EC2,  
>  
> I would like to dynamically resizing the cluster so as to reduce the cost, is there any solution to achieve this?  
>  
> E.g. I would like to cut the cluster size with a half, is it safe to just shutdown the instances (if some tasks are just running on them, can I rely on the speculative execution to re-run them on other nodes?)
>  
> I cannot use EMR, since I’m running a customized version of Hadoop  
>  
> Best,  
>  
> --  
> Nan Zhu
> School of Computer Science,
> McGill University
>