Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: when Standby Namenode is doing checkpoint, the Active NameNode is slow.


Copy link to this message
-
Re: when Standby Namenode is doing checkpoint, the Active NameNode is slow.
Hi,

I'm agreed with Harsh comment on image file compression and transfer
bandwidth parameter for optimizing checkpoint process.

In addition I'm not able to correlate your performance program log
timings(less then 10) and file transfer logs timing on active/stand by
nodes.

Thanks
On Tue, Aug 13, 2013 at 4:06 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Perhaps turning on fsimage compression may help. See
> documentation of dfs.image.compress at
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> .
>
> You can also try to throttle the bandwidth it uses via
> dfs.image.transfer.bandwidthPerSec.
>
> On Tue, Aug 13, 2013 at 3:45 PM, lei liu <[EMAIL PROTECTED]> wrote:
> > I write one programm to test NameNode performance. Please see the
> > EditLogPerformance.java
> >
> > I use 60 threads to execute the EditLogPerformance.javacode,  the testing
> > result is below content:
> > 2013-08-13 17:43:01,479 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10392810  speed:1055
> > 2013-08-13 17:43:11,482 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10407310  speed:725
> > 2013-08-13 17:43:21,484 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10407358  speed:2
> > 2013-08-13 17:43:31,487 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10407490  speed:6
> > 2013-08-13 17:43:41,490 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10407624  speed:6
> > 2013-08-13 17:43:51,493 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10408690  speed:53
> > 2013-08-13 17:44:01,496 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10422220  speed:676
> > 2013-08-13 17:44:11,499 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10445216  speed:1149
> > 2013-08-13 17:44:21,502 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10465166  speed:997
> > 2013-08-13 17:44:31,505 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10486614  speed:1072
> > 2013-08-13 17:44:41,508 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10506778  speed:1008
> > 2013-08-13 17:44:51,511 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10526660  speed:994
> > 2013-08-13 17:45:01,514 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10548092  speed:1071
> > 2013-08-13 17:45:11,517 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10569892  speed:1090
> > 2013-08-13 17:45:21,520 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10593296  speed:1170
> > 2013-08-13 17:45:31,523 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10614478  speed:1059
> > 2013-08-13 17:45:41,526 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10636006  speed:1076
> > 2013-08-13 17:45:51,529 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10656430  speed:1021
> > 2013-08-13 17:46:01,532 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10677774  speed:1067
> > 2013-08-13 17:46:11,534 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10699096  speed:1066
> > 2013-08-13 17:46:21,537 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10720970  speed:1093
> > 2013-08-13 17:46:31,540 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10741432  speed:1023
> > 2013-08-13 17:46:41,543 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10760854  speed:971
> > 2013-08-13 17:46:51,546 INFO  my.EditLogPerformance
> > (EditLogPerformance.java:run(37)) - totalCount:10781680  speed:1041
> > 2013-08-13 17:47