Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - delete rows without writing HLog may be appear in the future?


Copy link to this message
-
Re: delete rows without writing HLog may be appear in the future?
ramkrishna vasudevan 2012-11-21, 17:00
Are you doing lot of moves and also major compaction is happening ?

See HBASE-6059.  That could be a reason. Ideally it should have been
committed to 0.94.  Which was not done? Am i missing something?

Regards
Ram

On Wed, Nov 21, 2012 at 9:55 PM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> Kevin,  Yes I agree with no of WALs kept.  Lowering it will allow frequent
> flushes and that should lower the occurence of this issue.
> The more the flushes happen the lesser the probability of this issue
> occurence.
>
> But still if a RS crashes after doing the WAL less deletes then the puts
> are bound to reappear.  That is why i thought of the CP based approach. Pls
> correct me if i was wrong.
>
> Regards
> RAm
>
>
>
> On Wed, Nov 21, 2012 at 9:13 PM, Kevin O'dell <[EMAIL PROTECTED]>wrote:
>
>> I would recommend delete with HLog put, but lets say your writes are
>> minimal you should only have 32 hours(default) of WAL around at a time
>> before the they are all flushed from too many HLogs.  So you should not
>> have week old deletes coming back.  One thought would be to raise your WAL
>> size but lower the total number WALs kept.  This will retain the same
>> amount of data but should flush fully quicker.
>>
>> Can someone thought check the above logic?
>>
>> On Wed, Nov 21, 2012 at 10:37 AM, Bing Jiang <[EMAIL PROTECTED]
>> >wrote:
>>
>> > In our apps, deletes will be frequent, and it occurs to each records
>> every
>> > time, if write hlog, the performance and response will be low. In
>> fact,we
>> > can bear with some records with delete fail, but recently I have found
>> more
>> > records delete some time ago, for example, one week , they reappear
>> > again.Then, that makes me curious about what should do next., delete
>> with
>> > writing hlog, or put without hlog....
>> > On Nov 21, 2012 11:19 PM, "Kevin O'dell" <[EMAIL PROTECTED]>
>> wrote:
>> >
>> > > Bing,
>> > >
>> > >   I am curious to hear more about Mike's question.  Why are you not
>> using
>> > > the WAL for your deletes?
>> > >
>> > > On Wed, Nov 21, 2012 at 10:17 AM, Bing Jiang <
>> [EMAIL PROTECTED]
>> > > >wrote:
>> > >
>> > > > yes,hbase has made a compaction between batch-put and deletes. any
>> > ideas?
>> > > >
>> > > > On Nov 21, 2012 11:10 PM, "Michael Segel" <
>> [EMAIL PROTECTED]>
>> > > > wrote:
>> > > > >
>> > > > > Some time later?
>> > > > >
>> > > > > Time of course is relative, so I have to ask what occurred between
>> > the
>> > > > write and the delete?
>> > > > > How much time? Did you have any compactions in between the write
>> and
>> > > the
>> > > > delete?
>> > > > >
>> > > > > Why are you not consistent in your use of the WAL ?
>> > > > >
>> > > > >
>> > > > > On Nov 21, 2012, at 6:37 AM, Bing Jiang <[EMAIL PROTECTED]
>> >
>> > > > wrote:
>> > > > >
>> > > > > > hi,all.
>> > > > > > I want to describe a phenomenon that happens to our hbase
>> cluster.
>> > > > > > I use puts(List<Put>) to insert many records with writing hlog
>> > > enable,
>> > > > > > and some time later I delete all of these records with writing
>> hlog
>> > > > disable.
>> > > > > > When one week later, i scan the table, I found some records I
>> have
>> > > > delete
>> > > > > > reappear again.
>> > > > > > It is an interesting case. In my opinion, if we delete data
>> without
>> > > > enable
>> > > > > > writing hlog, when regionserver fails, the log will replay in
>> > another
>> > > > > > regionserver.
>> > > > > > Can anyone tell me if I persist on deleting records without
>> enable
>> > > > writing
>> > > > > > hlog, is there a way to prevent these records from reappearing
>> > again
>> > > > some
>> > > > > > time later?
>> > > > > >
>> > > > > > Cheers!
>> > > > > > --
>> > > > > > Bing Jiang
>> > > > > > weibo: http://weibo.com/jiangbinglover
>> > > > > > BLOG: http://blog.sina.com.cn/jiangbinglover
>> > > > > > BLOG: http://www.binospace.com
>> > > > > > National Research Center for Intelligent Computing Systems