Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Risk about RS logs clean ?


Copy link to this message
-
Re: Risk about RS logs clean ?
Bing Jiang 2013-05-23, 01:27
Hi,Sergey.
The version of hbase in our environment is 0.94.3, and the FSHLog.java
comes from 0.95 or version above.
And it adds such codes in FSHLog::cleanOldLogs,
 long oldestOutstandingSeqNum = Long.MAX_VALUE;
    synchronized (oldestSeqNumsLock) {
      Long oldestFlushing = (oldestFlushingSeqNums.size() > 0)
        ? Collections.min(oldestFlushingSeqNums.values()) : Long.MAX_VALUE;
      Long oldestUnflushed = (oldestUnflushedSeqNums.size() > 0)
        ? Collections.min(oldestUnflushedSeqNums.values()) : Long.MAX_VALUE;
      oldestOutstandingSeqNum = Math.min(oldestFlushing, oldestUnflushed);
    }

Which is different from the function from 0.94.3.

 private byte [][] cleanOldLogs() throws IOException {
      Long oldestOutstandingSeqNum = getOldestOutstandingSeqNum();
  ...
  }
 private Long getOldestOutstandingSeqNum() {
    return Collections.min(this.lastSeqWritten.values());
  }

And I think the version in trunk is safe.

Thanks for Sergey.
2013/5/23 Sergey Shelukhin <[EMAIL PROTECTED]>

> FSHLog (in trunk) stores the earliest seqnums for each region in current
> memstore, and earliest flushing seqnum (see
> FSHLog::start/complete/abortCacheFlush). When logs are deleted the logs
> with seqnums that are above the earliest flushing/flushed seqnum for any
> region are not deleted (see FSHLog::cleanOldLogs).
>
> On Wed, May 22, 2013 at 5:39 AM, Bing Jiang <[EMAIL PROTECTED]
> >wrote:
>
> > Hi,all
> > I want to know how RS eliminates the unnecessary hlogs.
> > lastSeqNum stores <RegionName, latest KV Seq id>
> > and
> > outputfiles stores <last Seq id before new hlog file, file path>
> >
> > So, how does rs guarantee that the kv in the hlog to be cleared  have
> been
> > already flushed from memstore into hfile.
> > I have try to read source code to make sense, however, I am not sure
> > whether it is a source of the risk of data loss.
> >
> > Thanks.
> > --
> > Bing Jiang
> > Tel:(86)134-2619-1361
> > weibo: http://weibo.com/jiangbinglover
> > BLOG: http://blog.sina.com.cn/jiangbinglover
> > National Research Center for Intelligent Computing Systems
> > Institute of Computing technology
> > Graduate University of Chinese Academy of Science
> >
>

--
Bing Jiang
Tel:(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science