Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: A question of QJM with HDFS federation


Copy link to this message
-
Re: A question of QJM with HDFS federation
Azuryy Yu 2013-04-15, 03:30
Hi Harsh,

If they are two separate cluster, instead of federated, haing the same
cluster ID, but using different name service ID, so can they use the same
journal nodes and ZK nodes?

and if they are also separated cluster, with different clusterID, but using
the same name service ID, so can they use the same journal nodes and ZK
nodes?

Thanks.
On Mon, Apr 8, 2013 at 2:57 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:

> Thank you very much, Harsh.
>
> not yet question now.
>
> --Send from my Sony mobile.
> On Apr 8, 2013 2:51 PM, "Harsh J" <[EMAIL PROTECTED]> wrote:
>
>> Hi Azurry,
>>
>> QJM:
>>
>> Yes, multiple nameservices can share a single QJM set. The QJM
>> configuration allows for a journal ID prefix path which you should
>> configure to be the nameservice ID. You do not need to change disk
>> paths/etc. at all.
>>
>> For example,
>>
>> NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
>> qjournal://node1.example.com:8485;node2.example.com:8485;
>> node3.example.com:8485/NS1
>>
>> NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
>> qjournal://node1.example.com:8485;node2.example.com:8485;
>> node3.example.com:8485/NS2
>>
>> Which will separate both of them logically and still make them use the
>> same QJM set of nodes.
>>
>> ZKFC:
>>
>> Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
>> NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
>> the same ha.zookeeper.quorum value since the ZKFC automatically reuses
>> the nameservice ID as its parent znode name on the ZK instance, and
>> won't collide with another NSs' ZKFCs.
>>
>> Do post back if there are still some more doubts.
>>
>> On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>> > Hi dears,
>> > I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>> >
>> > 1) if we also configured HDFS federation,  such as:
>> >
>> >     NN1 is active, NN2 is standby
>> >     NN3 is active, NN4 is standby
>> >
>> > they are configured as HDFS federation, then,
>> > Can these four NNs using the same Journal nodes and ZKs?
>> >
>> > if your answer is yes, does that enough just using different
>> > dfs.journalnode.edits.dir?
>> >
>> > such as:
>> > NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
>> > NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
>> >
>> >
>> > Thanks.
>>
>>
>>
>> --
>> Harsh J
>>
>