Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> 答复: why my test result on dfs short circuit read is slower?


Copy link to this message
-
答复: why my test result on dfs short circuit read is slower?
I'm not very clear about your senario, just a kindly reminder: "If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general",  see HDFS-2246's release note for more info
If you didn't enable security at all, please ignore my comments:)

Regards,
Liang
________________________________________
发件人: Liu, Raymond [[EMAIL PROTECTED]]
发送时间: 2013年2月16日 11:40
收件人: [EMAIL PROTECTED]
主题: RE: why my test result on dfs short circuit read is slower?

Hi Liang

Did you mean set dfs.permissions to false?

Is that all I need to do to disable security feature? Cause It seems to me that without change dfs.block.local-path-access.user, dfs.permissions alone doesn't works. HBASE still fall back to go through datanode to read data.
>
> Hi Raymond,
>
> did you enable security feature in your cluster?  there'll be no obvious benefit
> be found if so.
>
> Regards,
> Liang
> _______________________________________
> 发件人: Liu, Raymond [[EMAIL PROTECTED]]
> 发送时间: 2013年2月16日 11:10
> 收件人: [EMAIL PROTECTED]
> 主题: why my test result on dfs short circuit read is slower?
>
> Hi
>
>         I tried to use short circuit read to improve my hbase cluster MR scan
> performance.
>
>         I have the following setting in hdfs-site.xml
>
>         dfs.client.read.shortcircuit set to true
>         dfs.block.local-path-access.user set to MR job runner.
>
>         The cluster is 1+4 node and each data node have 16cpu/4HDD, with
> all hbase table major compact thus all data is local.
>         I have hoped that the short circuit read will improve the
> performance.
>
>         While the test result is that with short circuit read enabled, the
> performance actually dropped 10-15%. Say scan a 50G table cost around 100s
> instead of 90s.
>
>         My hadoop version is 1.1.1, any idea on this? Thx!
>
> Best Regards,
> Raymond Liu
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB