Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> some ideas for QJM and NFS


Copy link to this message
-
Re: 答复: 答复: some ideas for QJM and NFS
I don't think this is an issue. QJM talks with JN  using RPC,  the default
handlers are enough for both DN and JN in my testing.
On Mon, Feb 18, 2013 at 4:02 PM, 谢良 <[EMAIL PROTECTED]> wrote:

>  IMHO, it's not a recommented deploy manner to deploy JN and DN into same
> nodes.
>
> Regards,
> Liang
>   ------------------------------
> *发件人:* Azuryy Yu [[EMAIL PROTECTED]]
> *发送时间:* 2013年2月18日 15:56
> *收件人:* [EMAIL PROTECTED]
> *主题:* Re: 答复: some ideas for QJM and NFS
>
>   All JNs are deployed on the same node with DN.
>
>
> On Mon, Feb 18, 2013 at 3:35 PM, 谢良 <[EMAIL PROTECTED]> wrote:
>
>>  Hi Azuryy, just want to confirm one thing, your JN did not deploy on
>> the same machines within DN, right ?
>>
>> Regards,
>> Liang
>>   ------------------------------
>> *发件人:* Azuryy Yu [[EMAIL PROTECTED]]
>> *发送时间:* 2013年2月18日 15:22
>> *收件人:* [EMAIL PROTECTED]
>> *主题:* Re: some ideas for QJM and NFS
>>
>>    Hi,
>>
>> I did it on hadoop-2.0.3-alpha without HA as following:
>>
>> [root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hdfs dfs -put
>> testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S
>> 2013-02-18_15:20:01
>> 13/02/18 15:20:02 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> 2013-02-18_15:20:30
>>
>>  so the performance is a little bit better than hadoop-1.0.4.
>>
>>
>>
>> On Mon, Feb 18, 2013 at 2:53 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>>
>>> Oh, yes, you are right, George. I'll probably do it in the next days.
>>>
>>>
>>> On Mon, Feb 18, 2013 at 2:47 PM, George Datskos <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>>  Hi Azuryy,
>>>>
>>>> So you have measurements for hadoop-1.0.4 and hadoop-2.0.3+QJM, but I
>>>> think you should also measure hadoop-2.0.3 _wihout_ QJM so you can know for
>>>> sure if the performance degrade is actually related to QJM or not.
>>>>
>>>>
>>>> George
>>>>
>>>>
>>>>    Hi,
>>>>
>>>>  HarshJ is a good guy, I've seen this JIRA:
>>>> https://issues.apache.org/jira/browse/HDFS-4508
>>>>
>>>>  I have a test cluster hadoop-1.0.4, I've upgrade to
>>>> hadoop-2.0.3-alpha. mu cluster is very small, four nodes totally.
>>>>
>>>>  then I did some test on the original Hadoop and new Hadoop, the
>>>> testing is very simple: I have a data file with 450MB, I just put it on the
>>>> HDFS.
>>>>
>>>>  block size: 128MB, replica: 2
>>>>
>>>>  the following is the result:
>>>>
>>>> [root@webdm test]# ll testspeed.tar.gz
>>>> -rw-r--r-- 1 root root 452M Feb 18 13:54 testspeed.tar.gz
>>>> [root@webdm test]#
>>>>
>>>>  //On the hadoop-1.0.4
>>>>  [root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hadoop dfs -put
>>>> testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S
>>>> 2013-02-18_13:54:24
>>>> Warning: $HADOOP_HOME is deprecated.
>>>> 2013-02-18_13:54:58
>>>>
>>>>  //On the hadoop-2.0.3-alpha with QJM
>>>>  [root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hdfs dfs -put
>>>> testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S
>>>> 2013-02-18_14:13:29
>>>> 13/02/18 14:13:30 WARN util.NativeCodeLoader: Unable to load
>>>> native-hadoop library for your platform... using builtin-java classes where
>>>> applicable
>>>> 2013-02-18_14:14:33
>>>>
>>>>  I do think QJM HA feature affect the performance, because each writer
>>>> from QJM, it will do: fence old writer; sync in-progress log; start new log
>>>> segment; then write. only if writer received a successful response from a
>>>> quorum of JNs, writer finished for this time.
>>>>
>>>>  But NFS HA just write log segment in the local and NFS, when it
>>>> receive successful response from NFS, it finished this time.
>>>>
>>>>  So, I just suggest we always keep these two HA features in future,
>>>> even in the stable release. which one should be used, which depends on
>>>> yourself based on your infrastructure.
>>>>
>>>>  Thanks.
>>>>
>>>>
>>>>
>>>
>>
>