Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - Question for edit log retain


Copy link to this message
-
Re: Question for edit log retain
Aaron T. Myers 2013-07-01, 23:02
Hi Azuryy,

On Wed, Jun 26, 2013 at 6:12 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:

> Hi Dear all,
>
> I have some confusion for edit log retain,
>
> NNStorageRetentionManager.java:
>
> 1)
> purgeCheckpointsOlderThan()
>   What's mean of check point here?
>

Here, "checkpoint" is referring to the fsimage files on the NN.
>
> 2)purgeOldStorage()
>   I cannot understand the calculation of minimum txid, I think I can
> understand it if I know these keys indicates.
> DFS_NAMENODE_NUM_CHECKPOINTS_RETAINED_KEY
>

This is the number of old fsimage files to retain on the NN during purging.
> DFS_NAMENODE_NUM_EXTRA_EDITS_RETAINED_KEY
>

This is the number of extra transactions to retain during purging. Here,
"extra" is referring to transactions beyond what is strictly required to
play back all FS changes since the last checkpoint (fsimage.)
> DFS_NAMENODE_MAX_EXTRA_EDITS_SEGMENTS_RETAINED_KEY
>

This is the maximum number of extra edit _segments_ to retain during
purging. Segments refer to finalized edit log files, with a start and end
transaction ID. A single edit file is a "segment."
>
> Can any developer give me a short detail explanation? Thanks much.
>