Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: pauses during startup (maybe network related?)


Copy link to this message
-
Re: pauses during startup (maybe network related?)
Hi Ted,

2013-05-23 19:28:19,937 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
...
2013-05-23 19:28:26,801 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 28 on 9000: starting

There are a couple of relevant activities that happen during namenode
startup in between these 2 log statements.  It loads the current fsimage
(persistent copy of file system metadata), merges in the edits log
(transaction log containing all file system metadata changes since the last
checkpoint), and then saves back a new fsimage file after that merge.
 Current versions of the Hadoop codebase will print some information to
logs about the volume of activity during this checkpointing process, so I
recommend looking for that in your logs to see if this explains it.
 Depending on whether or not your have a large number of transactions
queued since your last checkpoint, this whole process can cause namenode
startup to take several minutes.

If this becomes a regular problem, then you can run SecondaryNameNode or
BackupNode to perform periodic checkpoints in addition to the checkpoint
that occurs on namenode restart.  This is probably overkill for a dev
environment on your laptop though.

Hope this helps,

Chris Nauroth
Hortonworks
http://hortonworks.com/

On Thu, May 23, 2013 at 2:49 AM, Ted <[EMAIL PROTECTED]> wrote:

> Hi I'm running hadoop on my local laptop for development and
> everything "works" but there's some annoying pauses during the startup
> which causes the entire hadoop startup process to take up to 4 minutes
> and I'm wondering what it is and if I can do anything about it.
>
> I'm running everything on 1 machines, on fedora linux, hadoop-1.1.2,
> oracle jkd1.7.0_17, the machine is a dual core i5, and I have 8gb of
> ram and an SSD so it shouldn't be slow.
>
> When the system pauses, there is no cpu usage, no disk usage and no
> network usage (although I suspect it's waiting for the network to
> resolve or return something).
>
> Here's some snippets from the namenode logs during startup where you
> can see it just pauses for around 30 seconds or more with out errors
> or anything :
>
> ...
> 2013-05-23 19:26:37,660 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-05-23 19:26:37,676 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-05-23 19:27:54,144 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-05-23 19:27:54,144 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system started
> ...
> 2013-05-23 19:27:54,341 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: The
> dfs.support.append option is in your configuration, however append is
> not supported. This configuration option is no longer required to
> enable sync.
> 2013-05-23 19:27:54,341 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
> accessTokenLifetime=0 min(s)
> 2013-05-23 19:28:19,918 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStateMBean and NameNodeMXBean
> 2013-05-23 19:28:19,937 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> ...
> 2013-05-23 19:28:26,801 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 28 on 9000: starting
> 2013-05-23 19:28:26,833 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 31 on 9000: starting
> 2013-05-23 19:30:10,644 INFO org.apache.hadoop.hdfs.StateChange:
> BLOCK* NameSystem.registerDatanode: node registration from
> 127.0.0.1:50010 storage DS-651015167-192.168.1.5-50010-1369140176513
> 2013-05-23 19:30:10,650 INFO org.apache.hadoop.net.NetworkTopology:
> Adding a new node: /default-rack/127.0.0.1:50010
>
>
> I already start the system with : export
> HADOOP_OPTS="-Djava.net.preferIPv4Stack=true"