Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> How to speed up Hadoop?


Copy link to this message
-
Re: How to speed up Hadoop?
On 9/5/2013 8:57 PM, Preethi Vinayak Ponangi wrote:
> Solution 1: Throw more hardware at the cluster. That's the whole point
> of hadoop.
> Solution 2: Try to optimize the mapreduce jobs. It depends on what
> kind of jobs you are running.
>
> I wouldn't suggest decreasing the number of replications as it kind of
> defeats the purpose of using Hadoop. You could do this if you can't
> get more hardware, are running experimental non-critical
> non-production data.
>
> What kind of Hadoop monitoring are you talking about?
>
> Regards,
> Vinayak.
>
>
> On Thu, Sep 5, 2013 at 7:51 PM, Chris Embree <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     I think you just went backwards.   more replicas (generally
>     speaking) are better.
>
>     I'd take 60 cheap, 1 U servers over 20 "highly fault tolerant"
>     ones for almost every problem.  I'd get them for the same or less
>     $ too.
>
>
>
>
>     On Thu, Sep 5, 2013 at 8:41 PM, Sundeep Kambhampati
>     <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>     wrote:
>
>         Hi all,
>
>             I am looking for ways to configure Hadoop inorder to speed
>         up data processing. Assuming all my nodes are highly fault
>         tolerant, will making data replication factor 1 speed up the
>         processing? Are there some way to disable failure monitoring
>         done by Hadoop?
>
>         Thank you for your time.
>
>         -Sundeep
>
>
>
Thank you your inputs. I can't currently add more hardware.

By monitoring I mean something like speculative execution.

Regards,
Sundeep
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB