Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - A question of QJM with HDFS federation


+
Azuryy Yu 2013-04-08, 05:23
Copy link to this message
-
Re: A question of QJM with HDFS federation
Harsh J 2013-04-08, 06:50
Hi Azurry,

QJM:

Yes, multiple nameservices can share a single QJM set. The QJM
configuration allows for a journal ID prefix path which you should
configure to be the nameservice ID. You do not need to change disk
paths/etc. at all.

For example,

NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/NS1

NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/NS2

Which will separate both of them logically and still make them use the
same QJM set of nodes.

ZKFC:

Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
the same ha.zookeeper.quorum value since the ZKFC automatically reuses
the nameservice ID as its parent znode name on the ZK instance, and
won't collide with another NSs' ZKFCs.

Do post back if there are still some more doubts.

On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
> Hi dears,
> I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>
> 1) if we also configured HDFS federation,  such as:
>
>     NN1 is active, NN2 is standby
>     NN3 is active, NN4 is standby
>
> they are configured as HDFS federation, then,
> Can these four NNs using the same Journal nodes and ZKs?
>
> if your answer is yes, does that enough just using different
> dfs.journalnode.edits.dir?
>
> such as:
> NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
> NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
>
>
> Thanks.

--
Harsh J
+
Azuryy Yu 2013-04-08, 06:57