Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: why my test result on dfs short circuit read is slower?


+
Harsh J 2013-02-16, 05:44
+
Liu, Raymond 2013-02-16, 05:53
+
Arpit Gupta 2013-02-16, 06:05
+
Liu, Raymond 2013-02-16, 06:22
+
Liu, Raymond 2013-02-16, 07:54
Copy link to this message
-
答复: why my test result on dfs short circuit read is slower?
I'm not very clear about your senario, just a kindly reminder: "If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general",  see HDFS-2246's release note for more info
If you didn't enable security at all, please ignore my comments:)

Regards,
Liang
________________________________________
发件人: Liu, Raymond [[EMAIL PROTECTED]]
发送时间: 2013年2月16日 11:40
收件人: [EMAIL PROTECTED]
主题: RE: why my test result on dfs short circuit read is slower?

Hi Liang

Did you mean set dfs.permissions to false?

Is that all I need to do to disable security feature? Cause It seems to me that without change dfs.block.local-path-access.user, dfs.permissions alone doesn't works. HBASE still fall back to go through datanode to read data.
>
> Hi Raymond,
>
> did you enable security feature in your cluster?  there'll be no obvious benefit
> be found if so.
>
> Regards,
> Liang
> _______________________________________
> 发件人: Liu, Raymond [[EMAIL PROTECTED]]
> 发送时间: 2013年2月16日 11:10
> 收件人: [EMAIL PROTECTED]
> 主题: why my test result on dfs short circuit read is slower?
>
> Hi
>
>         I tried to use short circuit read to improve my hbase cluster MR scan
> performance.
>
>         I have the following setting in hdfs-site.xml
>
>         dfs.client.read.shortcircuit set to true
>         dfs.block.local-path-access.user set to MR job runner.
>
>         The cluster is 1+4 node and each data node have 16cpu/4HDD, with
> all hbase table major compact thus all data is local.
>         I have hoped that the short circuit read will improve the
> performance.
>
>         While the test result is that with short circuit read enabled, the
> performance actually dropped 10-15%. Say scan a 50G table cost around 100s
> instead of 90s.
>
>         My hadoop version is 1.1.1, any idea on this? Thx!
>
> Best Regards,
> Raymond Liu