Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Hadoop hardware failure recovery


Copy link to this message
-
Re: Hadoop hardware failure recovery
On 13 August 2012 08:42, Harsh J <[EMAIL PROTECTED]> wrote:

> Hey Steve,
>
> Interesting, thanks for pointing that out! I didn't know that it
> disables this by default :)
>
>
It's always something to watch out for: someone implementing a disk FS, OS,
VM environment discovering that they get great benchmark numbers if they
make flushing async, and thinking "most people don't need it anyway".
That's mostly true -and some programs over-flush-, but if you do want to be
sure your data is saved to disk, these people are being dangerous rather
than helpful

I don't think it's an issue if you are saving to network mounted storage
-which can include the storage of the host OS. If you do some experiments,
NFS chat to the host system is usually as fast as working with a virtual
HDD in the same host OS -which has extra layers of indirection. Virtual
HDDs can get fragmented even when the VM thinks it's just allocated big
linear blocks -you need to defrag the virtual HDD then the physical disk
image to correct that.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB