Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: 【dfs.namenode.shared.edits.dir can support different NameServices】


Copy link to this message
-
Re: 【dfs.namenode.shared.edits.dir can support different NameServices】
Thanks for Azuryy's reply.

I used configuration:

<property>
  <name>dfs.namenode.shared.edits.dir.hbasecluster.hnn1</name>

<value>qjournal://GS-CIX-SEV0001:8485;GS-CIX-SEV0002:8485;GS-CIX-SEV0003:8485/hbasecluster</value>
</property>

<property>
 <name>dfs.namenode.shared.edits.dir.hbasecluster.hnn2</name>

<value>qjournal://GS-CIX-SEV0001:8485;GS-CIX-SEV0002:8485;GS-CIX-SEV0003:8485/hbasecluster</value>
</property>
<property>
  <name>dfs.namenode.shared.edits.dir.commoncluster.cnn1</name>

<value>qjournal://GS-CIX-SEV0001:8485;GS-CIX-SEV0002:8485;GS-CIX-SEV0003:8485/commoncluster</value>
</property>
<property>
  <name>dfs.namenode.shared.edits.dir.commoncluster.cnn2</name>

<value>qjournal://GS-CIX-SEV0001:8485;GS-CIX-SEV0002:8485;GS-CIX-SEV0003:8485/commoncluster</value>
</property>

 Start JournalNodes by hands. and everything goes well now.
2013/7/4 Azuryy Yu <[EMAIL PROTECTED]>

> Hi Bing,
> HA is not confilct with HDFS federation.
>
> for example, you have two name services: cluster1, cluster2,
>
> then,
>
> <property>
>     <name>dfs.namenode.shared.edits.dir</name>
>     <value>qjournal://n1.com:8485;n2.com:8485/cluster1</value>
> </property>
>
> <property>
>     <name>dfs.namenode.shared.edits.dir</name>
>     <value>qjournal://n1.com:8485;n2.com:8485/cluster2</value>
>  </property>
>
>
> On Thu, Jul 4, 2013 at 2:46 PM, Bing Jiang <[EMAIL PROTECTED]>wrote:
>
>> hi, Chris.
>> I have traced the source code, and I find this issue comes from
>> sbin/start-dfs.sh,
>>
>> SHARED_EDITS_DIR=$($HADOOP_PREFIX/bin/hdfs getconf -confKey
>> dfs.namenode.shared.edits.dir 2>&-)
>>
>> If I set the suffix on dfs.namenode.shared.edits.dir.[namespace id].[nn
>> id], it will get null. So please take consideration about HA on multiple
>> nameservice.
>> So, please change the way of launching JournalNodes in  shell scripts
>> (sbin/start-dfs.sh)
>>
>>
>> Thanks.
>>
>>
>> 2013/7/4 Chris Nauroth <[EMAIL PROTECTED]>
>>
>>> Hello Bing,
>>>
>>> I have not tested this configuration myself, but from reading the code
>>> it does appear that dfs.namenode.shared.edits.dir supports appending the
>>> nameservice ID to the key, so that you can specify different directories
>>> for different federated namenodes in a single hdfs-site.xml.
>>>
>>>
>>> https://github.com/apache/hadoop-common/blob/branch-2.0.5-alpha/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L147
>>>
>>> Hope this helps,
>>>
>>> Chris Nauroth
>>> Hortonworks
>>> http://hortonworks.com/
>>>
>>>
>>>
>>> On Wed, Jul 3, 2013 at 6:21 AM, Bing Jiang <[EMAIL PROTECTED]>wrote:
>>>
>>>> hi,all.
>>>> Using hadoop-2.0.5-alpha.
>>>> and I meet a problem that I should configure different
>>>> dfs.namenode.shared.edits.dir for different NameSpace.
>>>> So could dfs.namenode.shared.edits.dir item can support multiple
>>>> nameservies to avoid retaining multiple conf/hdfs-site.xml in different
>>>> Namenode?
>>>>
>>>> Thanks~
>>>>
>>>>
>>>>
>>>> --
>>>> Bing Jiang
>>>> Tel:(86)134-2619-1361
>>>> weibo: http://weibo.com/jiangbinglover
>>>> BLOG: http://blog.sina.com.cn/jiangbinglover
>>>> National Research Center for Intelligent Computing Systems
>>>> Institute of Computing technology
>>>> Graduate University of Chinese Academy of Science
>>>>
>>>
>>>
>>
>>
>> --
>> Bing Jiang
>> Tel:(86)134-2619-1361
>> weibo: http://weibo.com/jiangbinglover
>> BLOG: http://blog.sina.com.cn/jiangbinglover
>> National Research Center for Intelligent Computing Systems
>> Institute of Computing technology
>> Graduate University of Chinese Academy of Science
>>
>
>
--
Bing Jiang
Tel:(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB