Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> How to speed up Hadoop?


Copy link to this message
-
Re: How to speed up Hadoop?
How about this: http://hadoop.apache.org/docs/stable/vaidya.html
I've never tried it myself, i was just reading about it today.
On Thu, Sep 5, 2013 at 5:57 PM, Preethi Vinayak Ponangi <
[EMAIL PROTECTED]> wrote:

> Solution 1: Throw more hardware at the cluster. That's the whole point of
> hadoop.
> Solution 2: Try to optimize the mapreduce jobs. It depends on what kind of
> jobs you are running.
>
> I wouldn't suggest decreasing the number of replications as it kind of
> defeats the purpose of using Hadoop. You could do this if you can't get
> more hardware, are running experimental non-critical non-production data.
>
> What kind of Hadoop monitoring are you talking about?
>
> Regards,
> Vinayak.
>
>
> On Thu, Sep 5, 2013 at 7:51 PM, Chris Embree <[EMAIL PROTECTED]> wrote:
>
>> I think you just went backwards.   more replicas (generally speaking) are
>> better.
>>
>> I'd take 60 cheap, 1 U servers over 20 "highly fault tolerant" ones for
>> almost every problem.  I'd get them for the same or less $ too.
>>
>>
>>
>>
>> On Thu, Sep 5, 2013 at 8:41 PM, Sundeep Kambhampati <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi all,
>>>
>>>     I am looking for ways to configure Hadoop inorder to speed up data
>>> processing. Assuming all my nodes are highly fault tolerant, will making
>>> data replication factor 1 speed up the processing? Are there some way to
>>> disable failure monitoring done by Hadoop?
>>>
>>> Thank you for your time.
>>>
>>> -Sundeep
>>>
>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB