Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: dynamically resizing Hadoop cluster on AWS?


Copy link to this message
-
Re: dynamically resizing Hadoop cluster on AWS?
Thank you very much for replying and sorry for posting on the wrong list  

Best,  

--  
Nan Zhu
School of Computer Science,
McGill University

On Thursday, October 24, 2013 at 1:06 AM, Jun Ping Du wrote:

> Move to @user alias.
>  
> ----- Original Message -----
> From: "Jun Ping Du" <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> Sent: Wednesday, October 23, 2013 10:03:27 PM
> Subject: Re: dynamically resizing Hadoop cluster on AWS?
>  
> If only compute node (TaskTracker or NodeManager) in your instance, then decommission nodes and shutdown related EC2 instances should be fine although some finished/running tasks might need to be re-run automatically. If future, we would support gracefully decommission (tracked by YARN-914 and MAPREDUCE-5381) so that no tasks need to be rerun in this case (but need to wait a while).
>  
> Thanks,
>  
> Junping
>  
> ----- Original Message -----
> From: "Nan Zhu" <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> Sent: Wednesday, October 23, 2013 8:15:51 PM
> Subject: Re: dynamically resizing Hadoop cluster on AWS?
>  
> Oh, I’m not running HDFS in the instances, I use S3 to save data
>  
> --  
> Nan Zhu
> School of Computer Science,
> McGill University
>  
>  
>  
> On Wednesday, October 23, 2013 at 11:11 PM, Nan Zhu wrote:
>  
> > Hi, all  
> >  
> > I’m running a Hadoop cluster on AWS EC2,  
> >  
> > I would like to dynamically resizing the cluster so as to reduce the cost, is there any solution to achieve this?  
> >  
> > E.g. I would like to cut the cluster size with a half, is it safe to just shutdown the instances (if some tasks are just running on them, can I rely on the speculative execution to re-run them on other nodes?)
> >  
> > I cannot use EMR, since I’m running a customized version of Hadoop  
> >  
> > Best,  
> >  
> > --  
> > Nan Zhu
> > School of Computer Science,
> > McGill University
> >  
> >  
>  
>  
>  
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB