Hi Liang

Did you mean set dfs.permissions to false?

Is that all I need to do to disable security feature? Cause It seems to me that without change dfs.block.local-path-access.user, dfs.permissions alone doesn't works. HBASE still fall back to go through datanode to read data.
>
> Hi Raymond,
>
> did you enable security feature in your cluster?  there'll be no obvious benefit
> be found if so.
>
> Regards,
> Liang
> _______________________________________
> 发件人: Liu, Raymond [[EMAIL PROTECTED]]
> 发送时间: 2013年2月16日 11:10
> 收件人: [EMAIL PROTECTED]
> 主题: why my test result on dfs short circuit read is slower?
>
> Hi
>
>         I tried to use short circuit read to improve my hbase cluster MR scan
> performance.
>
>         I have the following setting in hdfs-site.xml
>
>         dfs.client.read.shortcircuit set to true
>         dfs.block.local-path-access.user set to MR job runner.
>
>         The cluster is 1+4 node and each data node have 16cpu/4HDD, with
> all hbase table major compact thus all data is local.
>         I have hoped that the short circuit read will improve the
> performance.
>
>         While the test result is that with short circuit read enabled, the
> performance actually dropped 10-15%. Say scan a 50G table cost around 100s
> instead of 90s.
>
>         My hadoop version is 1.1.1, any idea on this? Thx!
>
> Best Regards,
> Raymond Liu

Alright, I think in my sequence read scenario, it is possible that shortcircuit read is actually slower than read through datanode.

For, when read through datanode, FS read operation is done by datanode daemon, while data processing is done by client.
Thus when client is processing the data, data node could read data at the same time and write it to local socket for client to read.
It take very few time for client to read from local socket.

While when client read native FS directly, all the job is done by client. It will be blocked for more time to read data from native FS for process than from local socket.

Overall, when CPU is not bound, and data node always prepare further data for client(due to sequence read scenario), the result is that shortcircuit read is slower though it cost less CPU load.

The CPU/idle/IOWait load seems also justify my guess

= For a scan only job:
Read through datanode:  CPU : 30-35% / IOWait : ~50% / 300 second
Shortcircuit read:  CPU : 25-30% / IOWait : ~40% / 330 second
Short circuit read is 10% slower

= For a job do more calculation Read through datanode:  CPU : 80-90% / IOWait : ~5-10% / 190 seconds
Shortcircuit read:  CPU : ~90% / IOWait : ~2-3% / 160 second
Short circuit read is 15% faster.

So, short circuit read is not always faster, especially when CPU is not bound and read by sequence, it will be slower. This is the best explain I can get now.
Any thoughts?

Raymond

>
> I have try to tune io.file.buffer.size to 128K instead of 4K ShortCircuit read
> performance is still worse than read through datanode.
>
> I am start to wondering, does shortcircuit read really help under hadoop 1.1.1
> version?
> I google to find a few people mention they got 2x gain or so upon CDH etc. But I
> really can't find out what else I can do to make it even just catch up normal
> read path....
>
> >
> > It seems to me that, with short circuit read enabled, the
> > BlockReaderLocal read data in 512/4096 bytes unit(checksum check
> > enabled/skiped)
> >
> > While when It go through datanode, the BlockSender.sendChunks will
> > read and sent data in 64K bytes units?
> >
> > Is that true? And if so, won't it explain that read through datanode
> > will be faster? Since it read data in bigger block size.
> >
> > Best Regards,
> > Raymond Liu
> >
> >
> > > -----Original Message-----
> > > From: Liu, Raymond [mailto:[EMAIL PROTECTED]]
> > > Sent: Saturday, February 16, 2013 2:23 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: RE: why my test result on dfs short circuit read is slower?
> > >
> > > Hi Arpit Gupta
> > >
> > > Yes,  this way also confirms that short circuit read is enabled on my
> cluster.
> > >
> > > 13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true
> > >
> > > 13/02/16 14:07:34 DEBUG hdfs.DFSClient: New BlockReaderLocal for
> > > file
> > >
> >
> /mnt/DP_disk4/raymond/hdfs/data/current/subdir63/blk_-2736548898990727
> > > 638 of size 134217728 startOffset 0 length 134217728 short circuit
> > > checksum false
> > >
> > > So , any possibility that other setting might impact short circuit
> > > read to has worse performance than read through datanode?
> > >
> > > Raymond
> > >
> > >
> > >
> > > >Another way to check if short circuit read is configured correctly.
> > >
> > > >As the user who is configured for short circuit read issue the
> > > >following
> > > command on a node where you expect the data to be read locally.
> > >
> > > >export HADOOP_ROOT_LOGGER=debug,console; hadoop dfs -cat
> > > >/path/to/file_on_hdfs
> > >
> > > >On the console you should see something like "hdfs.DFSClient: New
> > > BlockReaderLocal for file...."
> > >
> > > >This would confirm that short circuit read is happening.
> > >
> > > --
> > > >Arpit Gupta
> > > >Hortonworks Inc.
> > > >http://hortonworks.com/
> > >
> > > On Feb 15, 2013, at 9:53 PM, "Liu, Raymond" <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >
> > > Hi Harsh
> > >
> > > Yes, I did set both of these. While not in hbase-site.xml but hdfs-site.xml.

Probably readahead played a key role on the first scenario(scan only job) ?  the default LONG_READ_THRESHOLD_BYTES(BlockSender.java) is 256k in current codebase, and ReadaheadPool takes effect on normal read path.
Regards,
Liang
________________________________________
发件人: Liu, Raymond [[EMAIL PROTECTED]]
发送时间: 2013年2月18日 14:04
收件人: [EMAIL PROTECTED]
主题: RE: why my test result on dfs short circuit read is slower?

Alright, I think in my sequence read scenario, it is possible that shortcircuit read is actually slower than read through datanode.

For, when read through datanode, FS read operation is done by datanode daemon, while data processing is done by client.
Thus when client is processing the data, data node could read data at the same time and write it to local socket for client to read.
It take very few time for client to read from local socket.

While when client read native FS directly, all the job is done by client. It will be blocked for more time to read data from native FS for process than from local socket.

Overall, when CPU is not bound, and data node always prepare further data for client(due to sequence read scenario), the result is that shortcircuit read is slower though it cost less CPU load.

The CPU/idle/IOWait load seems also justify my guess

= For a scan only job:
Read through datanode:  CPU : 30-35% / IOWait : ~50% / 300 second
Shortcircuit read:  CPU : 25-30% / IOWait : ~40% / 330 second
Short circuit read is 10% slower

= For a job do more calculation Read through datanode:  CPU : 80-90% / IOWait : ~5-10% / 190 seconds
Shortcircuit read:  CPU : ~90% / IOWait : ~2-3% / 160 second
Short circuit read is 15% faster.

So, short circuit read is not always faster, especially when CPU is not bound and read by sequence, it will be slower. This is the best explain I can get now.
Any thoughts?

Raymond

>
> I have try to tune io.file.buffer.size to 128K instead of 4K ShortCircuit read
> performance is still worse than read through datanode.
>
> I am start to wondering, does shortcircuit read really help under hadoop 1.1.1
> version?
> I google to find a few people mention they got 2x gain or so upon CDH etc. But I
> really can't find out what else I can do to make it even just catch up normal
> read path....
>
> >
> > It seems to me that, with short circuit read enabled, the
> > BlockReaderLocal read data in 512/4096 bytes unit(checksum check
> > enabled/skiped)
> >
> > While when It go through datanode, the BlockSender.sendChunks will
> > read and sent data in 64K bytes units?
> >
> > Is that true? And if so, won't it explain that read through datanode
> > will be faster? Since it read data in bigger block size.
> >
> > Best Regards,
> > Raymond Liu
> >
> >
> > > -----Original Message-----
> > > From: Liu, Raymond [mailto:[EMAIL PROTECTED]]
> > > Sent: Saturday, February 16, 2013 2:23 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: RE: why my test result on dfs short circuit read is slower?
> > >
> > > Hi Arpit Gupta
> > >
> > > Yes,  this way also confirms that short circuit read is enabled on my
> cluster.
> > >
> > > 13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true
> > >
> > > 13/02/16 14:07:34 DEBUG hdfs.DFSClient: New BlockReaderLocal for
> > > file
> > >
> >
> /mnt/DP_disk4/raymond/hdfs/data/current/subdir63/blk_-2736548898990727
> > > 638 of size 134217728 startOffset 0 length 134217728 short circuit
> > > checksum false
> > >
> > > So , any possibility that other setting might impact short circuit
> > > read to has worse performance than read through datanode?
> > >
> > > Raymond
> > >
> > >
> > >
> > > >Another way to check if short circuit read is configured correctly.
> > >
> > > >As the user who is configured for short circuit read issue the
> > > >following
> > > command on a node where you expect the data to be read locally.
> > >
> > > >export HADOOP_ROOT_LOGGER=debug,console; hadoop dfs -cat
> > > >/path/to/file_on_hdfs
> > >
> > > >On the console you should see something like "hdfs.DFSClient: New

On Feb 17, 2013, at 7:09 PM, "Liu, Raymond" <[EMAIL PROTECTED]> wrote:

> io.file.buffer.size
Drop this down to 64KB not 128KB.

You have 16 cpu which really means 8 cores and 4 disks.

Do you have Ganglia up and running?

I'll wager that you'll see a lot of wait cpu cycles in both cases.

Also how much memory do you have on each machine?
Tuning is going to be hardware specific and without really understanding what each parameter does, you can hurt performance.

Michael Segel  | (m) 312.755.9623

Segel and Associates
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com