Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - optimizing HDFS writes with replication=1

Copy link to this message
optimizing HDFS writes with replication=1
John Lilley 2014-01-22, 00:10
I am writing temp files to HDFS with replication=1, so I expect the blocks to be stored on the writing node.  Are there any tips, in general, for optimizing write performance to HDFS?  I use 128K buffers in the write() calls.  Are there any parameters that can be set  on the connection or in HDFS configuration to optimize this use pattern?