-Re: hadoop namenode recovery
Harsh J 2013-01-17, 13:38
A very slow NFS is certainly troublesome, and slows down the write
performance for every edit waiting to get logged to disk (there's a
logSync_avg_time metric that could be monitored for this), and therefore a
dedicated NFS mount is required if you are unwilling to use the proper
If the write fails (due to NFS client timeout, or unavailability), then as
I mentioned, NN removes it from the write queue and carries on with the
remaining disks. Use of soft mounted NFS is important here, otherwise (i.e.
hard mounts) the NFS itself will end up hanging the NN. On a soft mounted
NFS the write will not block forever.
On Thu, Jan 17, 2013 at 3:23 PM, randy <[EMAIL PROTECTED]> wrote:
> I've seen NFS get in a state many times where the mount is still there,
> but it can't be written to or accessed. What happens in that case? If the
> network is congested or slow, does that slow down the overall NN
> On 01/15/2013 11:14 PM, Harsh J wrote:
>> The NFS mount is to be soft-mounted; so if the NFS goes down, the NN
>> ejects it out and continues with the local disk. If auto-restore is
>> configured, it will re-add the NFS if its detected good again later.
>> On Wed, Jan 16, 2013 at 7:04 AM, randy <[EMAIL PROTECTED]
>> <mailto:[EMAIL PROTECTED]>> wrote:
>> What happens to the NN and/or performance if there's a problem with
>> the NFS server? Or the network?
>> On 01/14/2013 11:36 PM, Harsh J wrote:
>> Its very rare to observe an NN crash due to a software bug in
>> production. Most of the times its a hardware fault you should
>> worry about.
>> On 1.x, or any non-HA-carrying release, the best you can get to
>> safeguard against a total loss is to have redundant disk volumes
>> configured, one preferably over a dedicated remote NFS mount.
>> This way
>> the NN is recoverable after the node goes down, since you can
>> retrieve a
>> current copy from another machine (i.e. via the NFS mount) and
>> set a new
>> node up to replace the older NN and continue along.
>> A load balancer will not work as the NN is not a simple
>> webserver - it
>> maintains state which you cannot sync. We wrote HA-HDFS features
>> address the very concern you have.
>> If you want true, painless HA, branch-2 is your best bet at this
>> An upcoming 2.0.3 release should include the QJM based HA
>> features that
>> is painless to setup and very reliable to use (over other
>> options), and
>> works with commodity level hardware. FWIW, we've (my team and I)
>> supporting several users and customers who're running the 2.x
>> based HA
>> in production and other types of environments and it has been
>> stable in our experience. There are also some folks in the
>> running 2.x based HDFS for HA/else.
>> On Tue, Jan 15, 2013 at 6:55 AM, Panshul Whisper
>> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
>> <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>**
>> Is there a standard way to prevent the failure of Namenode
>> crash in
>> a Hadoop cluster?
>> or what is the standard or best practice for overcoming the
>> point failure problem of Hadoop.
>> I am not ready to take chances on a production server with
>> 2.0 Alpha release, which claims to have solved the problem.
>> there any other things I can do to either prevent the
>> failure or
>> recover from the failure in a very short time.
>> Thanking You,