Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - RE: why my test result on dfs short circuit read is slower?


Copy link to this message
-
答复: why my test result on dfs short circuit read is slower?
谢良 2013-02-18, 06:59
Probably readahead played a key role on the first scenario(scan only job) ?  the default LONG_READ_THRESHOLD_BYTES(BlockSender.java) is 256k in current codebase, and ReadaheadPool takes effect on normal read path.
Regards,
Liang
________________________________________
发件人: Liu, Raymond [[EMAIL PROTECTED]]
发送时间: 2013年2月18日 14:04
收件人: [EMAIL PROTECTED]
主题: RE: why my test result on dfs short circuit read is slower?

Alright, I think in my sequence read scenario, it is possible that shortcircuit read is actually slower than read through datanode.

For, when read through datanode, FS read operation is done by datanode daemon, while data processing is done by client.
Thus when client is processing the data, data node could read data at the same time and write it to local socket for client to read.
It take very few time for client to read from local socket.

While when client read native FS directly, all the job is done by client. It will be blocked for more time to read data from native FS for process than from local socket.

Overall, when CPU is not bound, and data node always prepare further data for client(due to sequence read scenario), the result is that shortcircuit read is slower though it cost less CPU load.

The CPU/idle/IOWait load seems also justify my guess

= For a scan only job:
Read through datanode:  CPU : 30-35% / IOWait : ~50% / 300 second
Shortcircuit read:  CPU : 25-30% / IOWait : ~40% / 330 second
Short circuit read is 10% slower

= For a job do more calculation Read through datanode:  CPU : 80-90% / IOWait : ~5-10% / 190 seconds
Shortcircuit read:  CPU : ~90% / IOWait : ~2-3% / 160 second
Short circuit read is 15% faster.

So, short circuit read is not always faster, especially when CPU is not bound and read by sequence, it will be slower. This is the best explain I can get now.
Any thoughts?

Raymond

>
> I have try to tune io.file.buffer.size to 128K instead of 4K ShortCircuit read
> performance is still worse than read through datanode.
>
> I am start to wondering, does shortcircuit read really help under hadoop 1.1.1
> version?
> I google to find a few people mention they got 2x gain or so upon CDH etc. But I
> really can't find out what else I can do to make it even just catch up normal
> read path....
>
> >
> > It seems to me that, with short circuit read enabled, the
> > BlockReaderLocal read data in 512/4096 bytes unit(checksum check
> > enabled/skiped)
> >
> > While when It go through datanode, the BlockSender.sendChunks will
> > read and sent data in 64K bytes units?
> >
> > Is that true? And if so, won't it explain that read through datanode
> > will be faster? Since it read data in bigger block size.
> >
> > Best Regards,
> > Raymond Liu
> >
> >
> > > -----Original Message-----
> > > From: Liu, Raymond [mailto:[EMAIL PROTECTED]]
> > > Sent: Saturday, February 16, 2013 2:23 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: RE: why my test result on dfs short circuit read is slower?
> > >
> > > Hi Arpit Gupta
> > >
> > > Yes,  this way also confirms that short circuit read is enabled on my
> cluster.
> > >
> > > 13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true
> > >
> > > 13/02/16 14:07:34 DEBUG hdfs.DFSClient: New BlockReaderLocal for
> > > file
> > >
> >
> /mnt/DP_disk4/raymond/hdfs/data/current/subdir63/blk_-2736548898990727
> > > 638 of size 134217728 startOffset 0 length 134217728 short circuit
> > > checksum false
> > >
> > > So , any possibility that other setting might impact short circuit
> > > read to has worse performance than read through datanode?
> > >
> > > Raymond
> > >
> > >
> > >
> > > >Another way to check if short circuit read is configured correctly.
> > >
> > > >As the user who is configured for short circuit read issue the
> > > >following
> > > command on a node where you expect the data to be read locally.
> > >
> > > >export HADOOP_ROOT_LOGGER=debug,console; hadoop dfs -cat
> > > >/path/to/file_on_hdfs
> > >
> > > >On the console you should see something like "hdfs.DFSClient: New