Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: some ideas for QJM and NFS


Copy link to this message
-
Re: some ideas for QJM and NFS
George Datskos 2013-02-18, 06:47
Hi Azuryy,

So you have measurements for hadoop-1.0.4 and hadoop-2.0.3+QJM, but I
think you should also measure hadoop-2.0.3 _wihout_ QJM so you can know
for sure if the performance degrade is actually related to QJM or not.
George
> Hi,
>
> HarshJ is a good guy, I've seen this JIRA:
> https://issues.apache.org/jira/browse/HDFS-4508
>
> I have a test cluster hadoop-1.0.4, I've upgrade to
> hadoop-2.0.3-alpha. mu cluster is very small, four nodes totally.
>
> then I did some test on the original Hadoop and new Hadoop, the
> testing is very simple: I have a data file with 450MB, I just put it
> on the HDFS.
>
> block size: 128MB, replica: 2
>
> the following is the result:
>
> [root@webdm test]# ll testspeed.tar.gz
> -rw-r--r-- 1 root root 452M Feb 18 13:54 testspeed.tar.gz
> [root@webdm test]#
>
> //On the hadoop-1.0.4
> [root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hadoop dfs -put
> testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S
> 2013-02-18_13:54:24
> Warning: $HADOOP_HOME is deprecated.
> 2013-02-18_13:54:58
>
> //On the hadoop-2.0.3-alpha with QJM
> [root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hdfs dfs -put
> testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S
> 2013-02-18_14:13:29
> 13/02/18 14:13:30 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable
> 2013-02-18_14:14:33
>
> I do think QJM HA feature affect the performance, because each writer
> from QJM, it will do: fence old writer; sync in-progress log; start
> new log segment; then write. only if writer received a successful
> response from a quorum of JNs, writer finished for this time.
>
> But NFS HA just write log segment in the local and NFS, when it
> receive successful response from NFS, it finished this time.
>
> So, I just suggest we always keep these two HA features in future,
> even in the stable release. which one should be used, which depends on
> yourself based on your infrastructure.
>
> Thanks.

+
Todd Lipcon 2013-02-20, 00:04