Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> delete rows without writing HLog may be appear in the future?


Copy link to this message
-
Re: delete rows without writing HLog may be appear in the future?
If I set hbase.hregion.preclose.flush.size to zero, can HBase guarantee
that when HRegionServer quit, whether It will execute the last flush.
As Lars says, the issue refers to memstore flush, I have check that the
default value of ‘hbase.hregion.preclose.flush.size’ is 1024 * 1024 * 5.
I think if under the circumstance that we cannot bear with the low
performance of delete enable hlog, set ‘hbase.hregion.preclose.flush.size’
to zero, is it another choice?
Thanks.

----
Bing

2012/11/22 lars hofhansl <[EMAIL PROTECTED]>

> Hi Bing,
>
> I think you are referring to a memstore flush.
> The HLog represents the set of changes that are in the memstore (in ram)
> but not in an HFile on disk, yet.
>
>
> I am pretty sure there is no flaw in the flush/compaction logic when it
> comes to deletes.
>
>
> If you do not write the deletes to the WAL and the RS crashes it is
> expected that deletes there were not flushed to disk are lost.
>
> (And there's also HBASE-6059, which in some case resurfaces deleted data
> even when it was flushed to the WAL).
>
>
> -- Lars
>
>
> ________________________________
>  From: Bing Jiang <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wednesday, November 21, 2012 8:36 PM
> Subject: Re: delete rows without writing HLog may be appear in the future?
>
> I think when compaction is intrigued, if the records has already flushed
> into hdfs, whether it is worthless to retain the Hlog before that
> timestamp.
> In other ways, for example, some rows are deleted, then it executes a
> compaction, at the same time , the rows do not exist. So the hlog before
> the timestamp of compaction is not useful, and we can drop these unused
> wal.
> This is view of my own,  please correct me if wrong.
> ---
> Bing
>
>
> 2012/11/22 ramkrishna vasudevan <[EMAIL PROTECTED]>
>
> > Sorry Bing.. am not much clear as what you suggest
> > 'One idea occurs to me why not check or restore wal when compaction
> > executes. If it does, hbase can drop some unused hlog'.
> >
> > Could you be more clear?  Are you trying to read the WAL while compaction
> > is going on?
> >
> > Regards
> > Ram
> >
> > On Thu, Nov 22, 2012 at 9:23 AM, Bing Jiang <[EMAIL PROTECTED]
> > >wrote:
> >
> > > In our hbase cluster, I test if delete records with hlog or without.
> > > Attachment is my my test.
> > > The result of test can testify why I make a decision of delete rows
> > > without hlog .
> > >
> > >
> > >
> > > 2012/11/22 Bing Jiang <[EMAIL PROTECTED]>
> > >
> > >> Thanks for all your suggestion and talk.
> > >> One idea occurs to me why not check or restore wal when compaction
> > >> executes. If it does, hbase can drop some unused hlog, I think that
> > will be
> > >> effective to the issue.
> > >> please correct me if I am wrong.
> > >>
> > >> ---Bing
> > >>
> > >> 2012/11/22 lars hofhansl <[EMAIL PROTECTED]>
> > >>
> > >>> I have it on my list of things to do to allow deferred WAL flush as a
> > >>> per operation option (right now it's a CF option).
> > >>> You really do not want to do anything with the WAL off. If you use
> > >>> deferred flush there is still a chance that this might happen (the RS
> > could
> > >>> die in the few seconds after a Delete before it is flushed to the
> > WAL), but
> > >>> it should be a rare occurrance.
> > >>>
> > >>>
> > >>> -- Lars
> > >>>
> > >>>
> > >>>
> > >>> ________________________________
> > >>>  From: Bing Jiang <[EMAIL PROTECTED]>
> > >>> To: [EMAIL PROTECTED]
> > >>> Sent: Wednesday, November 21, 2012 7:20 AM
> > >>> Subject: Re: delete rows without writing HLog may be appear in the
> > >>> future?
> > >>>
> > >>> we need to confirm that put must be safe,but deletes must be quick
> and
> > >>> low-latency.
> > >>> On Nov 21, 2012 11:10 PM, "Michael Segel" <[EMAIL PROTECTED]
> >
> > >>> wrote:
> > >>>
> > >>> > Some time later?
> > >>> >
> > >>> > Time of course is relative, so I have to ask what occurred between
> > the
> > >>> > write and the delete?

Bing Jiang
Tel:(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB