Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: why my test result on dfs short circuit read is slower?


Copy link to this message
-
Re: why my test result on dfs short circuit read is slower?
If you want HBase to leverage the shortcircuit, the DN config
"dfs.block.local-path-access.user" should be set to the user running
HBase (i.e. hbase, for example), and the hbase-site.xml should have
"dfs.client.read.shortcircuit" defined in all its RegionServers. Doing
this wrong could result in performance penalty and some warn-logging,
as local reads will be attempted but will begin to fail.

On Sat, Feb 16, 2013 at 8:40 AM, Liu, Raymond <[EMAIL PROTECTED]> wrote:
> Hi
>
>         I tried to use short circuit read to improve my hbase cluster MR scan performance.
>
>         I have the following setting in hdfs-site.xml
>
>         dfs.client.read.shortcircuit set to true
>         dfs.block.local-path-access.user set to MR job runner.
>
>         The cluster is 1+4 node and each data node have 16cpu/4HDD, with all hbase table major compact thus all data is local.
>         I have hoped that the short circuit read will improve the performance.
>
>         While the test result is that with short circuit read enabled, the performance actually dropped 10-15%. Say scan a 50G table cost around 100s instead of 90s.
>
>         My hadoop version is 1.1.1, any idea on this? Thx!
>
> Best Regards,
> Raymond Liu
>
>

--
Harsh J
+
Liu, Raymond 2013-02-16, 05:53
+
Arpit Gupta 2013-02-16, 06:05
+
Liu, Raymond 2013-02-16, 06:22
+
Liu, Raymond 2013-02-16, 07:54
+
谢良 2013-02-16, 04:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB