Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> performance: HLog flush to disk each thread, can we decrease IO calls?


+
Mingjian Deng 2011-06-29, 10:27
Copy link to this message
-
Re: performance: HLog flush to disk each thread, can we decrease IO calls?
Hey Mingjian,

This sounds like a good idea  Your patch didn't make it through. Would you mind either filing a JIRA and uploading your patch there or at least posting it to something like pastebin so we can take a look.

-Joey

On Jun 29, 2011, at 3:27, Mingjian Deng <[EMAIL PROTECTED]> wrote:

> Hi:
>     We found that the hlog sync to disk each time. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
>     Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
>
>     So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to
> disk once it gets "updateLock".
>
>     In my test, it can increase the write speed of 40%.
>
>     Just see the HLog.patch.
>    
+
Dhruba Borthakur 2011-06-29, 13:14
+
Mingjian Deng 2011-06-29, 14:39
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB