Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> delete rows without writing HLog may be appear in the future?


Copy link to this message
-
Re: delete rows without writing HLog may be appear in the future?
Are you doing lot of moves and also major compaction is happening ?

See HBASE-6059.  That could be a reason. Ideally it should have been
committed to 0.94.  Which was not done? Am i missing something?

Regards
Ram

On Wed, Nov 21, 2012 at 9:55 PM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> Kevin,  Yes I agree with no of WALs kept.  Lowering it will allow frequent
> flushes and that should lower the occurence of this issue.
> The more the flushes happen the lesser the probability of this issue
> occurence.
>
> But still if a RS crashes after doing the WAL less deletes then the puts
> are bound to reappear.  That is why i thought of the CP based approach. Pls
> correct me if i was wrong.
>
> Regards
> RAm
>
>
>
> On Wed, Nov 21, 2012 at 9:13 PM, Kevin O'dell <[EMAIL PROTECTED]>wrote:
>
>> I would recommend delete with HLog put, but lets say your writes are
>> minimal you should only have 32 hours(default) of WAL around at a time
>> before the they are all flushed from too many HLogs.  So you should not
>> have week old deletes coming back.  One thought would be to raise your WAL
>> size but lower the total number WALs kept.  This will retain the same
>> amount of data but should flush fully quicker.
>>
>> Can someone thought check the above logic?
>>
>> On Wed, Nov 21, 2012 at 10:37 AM, Bing Jiang <[EMAIL PROTECTED]
>> >wrote:
>>
>> > In our apps, deletes will be frequent, and it occurs to each records
>> every
>> > time, if write hlog, the performance and response will be low. In
>> fact,we
>> > can bear with some records with delete fail, but recently I have found
>> more
>> > records delete some time ago, for example, one week , they reappear
>> > again.Then, that makes me curious about what should do next., delete
>> with
>> > writing hlog, or put without hlog....
>> > On Nov 21, 2012 11:19 PM, "Kevin O'dell" <[EMAIL PROTECTED]>
>> wrote:
>> >
>> > > Bing,
>> > >
>> > >   I am curious to hear more about Mike's question.  Why are you not
>> using
>> > > the WAL for your deletes?
>> > >
>> > > On Wed, Nov 21, 2012 at 10:17 AM, Bing Jiang <
>> [EMAIL PROTECTED]
>> > > >wrote:
>> > >
>> > > > yes,hbase has made a compaction between batch-put and deletes. any
>> > ideas?
>> > > >
>> > > > On Nov 21, 2012 11:10 PM, "Michael Segel" <
>> [EMAIL PROTECTED]>
>> > > > wrote:
>> > > > >
>> > > > > Some time later?
>> > > > >
>> > > > > Time of course is relative, so I have to ask what occurred between
>> > the
>> > > > write and the delete?
>> > > > > How much time? Did you have any compactions in between the write
>> and
>> > > the
>> > > > delete?
>> > > > >
>> > > > > Why are you not consistent in your use of the WAL ?
>> > > > >
>> > > > >
>> > > > > On Nov 21, 2012, at 6:37 AM, Bing Jiang <[EMAIL PROTECTED]
>> >
>> > > > wrote:
>> > > > >
>> > > > > > hi,all.
>> > > > > > I want to describe a phenomenon that happens to our hbase
>> cluster.
>> > > > > > I use puts(List<Put>) to insert many records with writing hlog
>> > > enable,
>> > > > > > and some time later I delete all of these records with writing
>> hlog
>> > > > disable.
>> > > > > > When one week later, i scan the table, I found some records I
>> have
>> > > > delete
>> > > > > > reappear again.
>> > > > > > It is an interesting case. In my opinion, if we delete data
>> without
>> > > > enable
>> > > > > > writing hlog, when regionserver fails, the log will replay in
>> > another
>> > > > > > regionserver.
>> > > > > > Can anyone tell me if I persist on deleting records without
>> enable
>> > > > writing
>> > > > > > hlog, is there a way to prevent these records from reappearing
>> > again
>> > > > some
>> > > > > > time later?
>> > > > > >
>> > > > > > Cheers!
>> > > > > > --
>> > > > > > Bing Jiang
>> > > > > > weibo: http://weibo.com/jiangbinglover
>> > > > > > BLOG: http://blog.sina.com.cn/jiangbinglover
>> > > > > > BLOG: http://www.binospace.com
>> > > > > > National Research Center for Intelligent Computing Systems
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB