Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - question about ZKFC daemon


Copy link to this message
-
Re: question about ZKFC daemon
Colin McCabe 2013-01-14, 19:49
Hi ESGLinux,

In production, you need to run QJM on at least 3 nodes.  You also need
to run ZKFC on at least 3 nodes.  You can run them on the same nodes
if you like, though.

Of course, none of this is "needed" to set up an example cluster.  If
you just want to try something out, you can run everything on the same
node if you want.  It depends on what you're trying to do.

cheers,
Colin
On Fri, Dec 28, 2012 at 3:02 AM, ESGLinux <[EMAIL PROTECTED]> wrote:
> Thank you for your answer Craig,
>
> I´m planning my cluster and for now I´m not sure how many machines I need;-)
>
> If I have doubt i´ll what clouder say and If have a problem I have where to
> ask for explications :-)
>
> ESGLinux
>
>
>
> 2012/12/28 Craig Munro <[EMAIL PROTECTED]>
>>
>> OK, I have reliable storage on my datanodes so not an issue for me.  If
>> that's what Cloudera recommends then I'm sure it's fine.
>>
>> On Dec 28, 2012 10:38 AM, "ESGLinux" <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi Craig,
>>>
>>> I´m a bit confused, I have read this from cloudera:
>>> https://ccp.cloudera.com/display/CDH4DOC/Hardware+Configuration+for+Quorum-based+Storage
>>>
>>> The JournalNode daemon is relatively lightweight, so these daemons can
>>> reasonably be collocated on machines with other Hadoop daemons, for example
>>> NameNodes, the JobTracker, or the YARN ResourceManager.
>>> Cloudera recommends that you deploy the JournalNode daemons on the
>>> "master" host or hosts (NameNode, Standby NameNode, JobTracker, etc.) so the
>>> JournalNodes' local directories can use the reliable local storage on those
>>> machines.
>>> There must be at least three JournalNode daemons, since edit log
>>> modifications must be written to a majority of JournalNodes
>>>
>>> as you can read they recommend to put journalnode daemons with the
>>> namenodes, but you say the opposite.??¿?¿??
>>>
>>>
>>> Thanks for your answer,
>>>
>>> ESGLinux,
>>>
>>>
>>>
>>>
>>> 2012/12/28 Craig Munro <[EMAIL PROTECTED]>
>>>>
>>>> You need the following:
>>>>
>>>> - active namenode + zkfc
>>>> - standby namenode + zkfc
>>>> - pool of journal nodes (odd number, 3 or more)
>>>> - pool of zookeeper nodes (odd number, 3 or more)
>>>>
>>>> As the journal nodes hold the namesystem transactions they should not be
>>>> co-located with the namenodes in case of failure.  I distribute the journal
>>>> and zookeeper nodes across the hosts running datanodes or as Harsh says you
>>>> could co-locate them on dedicated hosts.
>>>>
>>>> ZKFC does not monitor the JobTracker.
>>>>
>>>> Regards,
>>>> Craig
>>>>
>>>> On Dec 28, 2012 9:25 AM, "ESGLinux" <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> well, If I have understand you I can configure my NN HA cluster this
>>>>> way:
>>>>>
>>>>> - Active NameNode + 1 ZKFC daemon + Journal Node
>>>>> - Standby NameNode + 1 ZKFC daemon + Journal Node
>>>>> - JobTracker node + 1 ZKFC daemon + Journal Node,
>>>>>
>>>>> Is this right?
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> ESGLinux,
>>>>>
>>>>> 2012/12/27 Harsh J <[EMAIL PROTECTED]>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> There are two different things here: Automatic Failover and Quorum
>>>>>> Journal Manager. The former, used via a ZooKeeper Failover Controller,
>>>>>> is to manage failovers automatically (based on health checks of NNs).
>>>>>> The latter, used via a set of Journal Nodes, is a medium of shared
>>>>>> storage for namesystem transactions that helps enable HA.
>>>>>>
>>>>>> In a typical deployment, you want 3 or more (odd) JournalNodes for
>>>>>> reliable HA, preferably on nodes of their own if possible (like you
>>>>>> would for typical ZooKeepers, and you may co-locate with those as
>>>>>> well) and one ZKFC for each NameNode (connected to the same ZK
>>>>>> quorum).
>>>>>>
>>>>>> On Thu, Dec 27, 2012 at 5:33 PM, ESGLinux <[EMAIL PROTECTED]> wrote:
>>>>>> > Hi all,
>>>>>> >
>>>>>> > I have a doubt about how to deploy the Zookeeper in a NN HA
>>>>>> > cluster,
>>>>>> >
>>>>>> > As far as I know, I need at least three nodes to run three ZooKeeper