Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Risk about RS logs clean ?


+
Bing Jiang 2013-05-22, 12:39
+
Sergey Shelukhin 2013-05-22, 17:53
+
Bing Jiang 2013-05-23, 01:27
Copy link to this message
-
Re: Risk about RS logs clean ?
Sergey Shelukhin 2013-05-23, 17:28
IIRC the version in previous branches should have an epic lock somewhere
(cacheFlushLock or something like that) that should make this map
manipulations safe also.

On Wed, May 22, 2013 at 6:27 PM, Bing Jiang <[EMAIL PROTECTED]>wrote:

> Hi,Sergey.
> The version of hbase in our environment is 0.94.3, and the FSHLog.java
> comes from 0.95 or version above.
> And it adds such codes in FSHLog::cleanOldLogs,
>  long oldestOutstandingSeqNum = Long.MAX_VALUE;
>     synchronized (oldestSeqNumsLock) {
>       Long oldestFlushing = (oldestFlushingSeqNums.size() > 0)
>         ? Collections.min(oldestFlushingSeqNums.values()) : Long.MAX_VALUE
> ;
>       Long oldestUnflushed = (oldestUnflushedSeqNums.size() > 0)
>         ? Collections.min(oldestUnflushedSeqNums.values()) : Long.
> MAX_VALUE;
>       oldestOutstandingSeqNum = Math.min(oldestFlushing, oldestUnflushed);
>     }
>
> Which is different from the function from 0.94.3.
>
>  private byte [][] cleanOldLogs() throws IOException {
>       Long oldestOutstandingSeqNum = getOldestOutstandingSeqNum();
>   ...
>   }
>  private Long getOldestOutstandingSeqNum() {
>     return Collections.min(this.lastSeqWritten.values());
>   }
>
> And I think the version in trunk is safe.
>
> Thanks for Sergey.
>
>
> 2013/5/23 Sergey Shelukhin <[EMAIL PROTECTED]>
>
>> FSHLog (in trunk) stores the earliest seqnums for each region in current
>> memstore, and earliest flushing seqnum (see
>> FSHLog::start/complete/abortCacheFlush). When logs are deleted the logs
>> with seqnums that are above the earliest flushing/flushed seqnum for any
>> region are not deleted (see FSHLog::cleanOldLogs).
>>
>> On Wed, May 22, 2013 at 5:39 AM, Bing Jiang <[EMAIL PROTECTED]
>> >wrote:
>>
>> > Hi,all
>> > I want to know how RS eliminates the unnecessary hlogs.
>> > lastSeqNum stores <RegionName, latest KV Seq id>
>> > and
>> > outputfiles stores <last Seq id before new hlog file, file path>
>> >
>> > So, how does rs guarantee that the kv in the hlog to be cleared  have
>> been
>> > already flushed from memstore into hfile.
>> > I have try to read source code to make sense, however, I am not sure
>> > whether it is a source of the risk of data loss.
>> >
>> > Thanks.
>> > --
>> > Bing Jiang
>> > Tel:(86)134-2619-1361
>> > weibo: http://weibo.com/jiangbinglover
>> > BLOG: http://blog.sina.com.cn/jiangbinglover
>> > National Research Center for Intelligent Computing Systems
>> > Institute of Computing technology
>> > Graduate University of Chinese Academy of Science
>> >
>>
>
>
>
> --
> Bing Jiang
> Tel:(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>