Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> optimizing HDFS writes with replication=1


Copy link to this message
-
optimizing HDFS writes with replication=1
I am writing temp files to HDFS with replication=1, so I expect the blocks to be stored on the writing node.  Are there any tips, in general, for optimizing write performance to HDFS?  I use 128K buffers in the write() calls.  Are there any parameters that can be set  on the connection or in HDFS configuration to optimize this use pattern?
Thanks
John
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB