-答复: why my test result on dfs short circuit read is slower?
谢良 2013-02-16, 04:04
I'm not very clear about your senario, just a kindly reminder: "If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general", see HDFS-2246's release note for more info
If you didn't enable security at all, please ignore my comments:)
发件人: Liu, Raymond [[EMAIL PROTECTED]]
发送时间: 2013年2月16日 11:40
收件人: [EMAIL PROTECTED]
主题: RE: why my test result on dfs short circuit read is slower?
Did you mean set dfs.permissions to false?
Is that all I need to do to disable security feature? Cause It seems to me that without change dfs.block.local-path-access.user, dfs.permissions alone doesn't works. HBASE still fall back to go through datanode to read data.
> Hi Raymond,
> did you enable security feature in your cluster? there'll be no obvious benefit
> be found if so.
> 发件人: Liu, Raymond [[EMAIL PROTECTED]]
> 发送时间: 2013年2月16日 11:10
> 收件人: [EMAIL PROTECTED]
> 主题: why my test result on dfs short circuit read is slower?
> I tried to use short circuit read to improve my hbase cluster MR scan
> I have the following setting in hdfs-site.xml
> dfs.client.read.shortcircuit set to true
> dfs.block.local-path-access.user set to MR job runner.
> The cluster is 1+4 node and each data node have 16cpu/4HDD, with
> all hbase table major compact thus all data is local.
> I have hoped that the short circuit read will improve the
> While the test result is that with short circuit read enabled, the
> performance actually dropped 10-15%. Say scan a 50G table cost around 100s
> instead of 90s.
> My hadoop version is 1.1.1, any idea on this? Thx!
> Best Regards,
> Raymond Liu