Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> What happened in hlog if data are deleted cuased by ttl?


Copy link to this message
-
Re: What happened in hlog if data are deleted cuased by ttl?
Sorry for that. I didn't use the right parameter. Now I get the point.

regards!

Yong

On Wed, Aug 22, 2012 at 10:49 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> Hey Yonghu,
>
> You are right that TTL "deletions" (it isn't exactly a delete, its
> more of a compact-time skip wizardry) do not go to the HLog as
> "events". Know that TTLs aren't applied "per-cell", they are applied
> on the whole CF globally. So there is no such thing as a TTL-write or
> a TTL-delete event. In fact, the Region-level Coprocessor too has no
> hooks for "TTL-events", as seen at
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html,
> for this doesn't happen on triggers.
>
> What you say about the compaction part is wrong however. Compaction
> too runs a regular store-file scanner to compact, and so does the
> regular Scan operation, to read (Both use the same file scanning
> mechanism/code). So there's no difference in how compact or a client
> scan handle TTL-expired row values from a store file, when reading it
> up.
>
> I also am not able to understand what your sample shell command list
> shows. As I see it, its shown that the HFile did have the entry in it
> after you had flushed it. Note that you mentioned the TTL at the CF
> level when creating the table, not in your "put" statement, and this
> is a vital point in understanding how TTLs work.
>
> On Wed, Aug 22, 2012 at 1:49 PM, yonghu <[EMAIL PROTECTED]> wrote:
>> I can fully understand normal deletion. But, in my point of view, ttl
>> deletion is different than the normal deletion. The insertion of ttl
>> data is recorded in hlog. But the ttl deletion is not recorded by
>> hlog. So, it failure occurs, should the ttl data be reinserted to data
>> or can we discard the certain ttl data? Moreover, ttl deletion is not
>> executed at data compaction time. Scanner needs to periodically scan
>> each Store file to execute deletion.
>>
>> regards!
>>
>> Yong
>>
>>
>>
>> On Tue, Aug 21, 2012 at 5:29 PM, jmozah <[EMAIL PROTECTED]> wrote:
>>> This helped me http://hadoop-hbase.blogspot.in/2011/12/deletion-in-hbase.html
>>>
>>>
>>> ./Zahoor
>>> HBase Musings
>>>
>>>
>>> On 14-Aug-2012, at 6:54 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Yonghu,
>>>>
>>>> A timestamp is stored along with each insert. The ttl is maintained at
>>>> the region-store level. Hence, when the log replays, all entries with
>>>> expired TTLs are automatically omitted.
>>>>
>>>> Also, TTL deletions happen during compactions, and hence do not
>>>> carry/need Delete events. When scanning a store file, TTL-expired
>>>> entries are automatically skipped away.
>>>>
>>>> On Tue, Aug 14, 2012 at 3:34 PM, yonghu <[EMAIL PROTECTED]> wrote:
>>>>> My hbase version is 0.92. I tried something as follows:
>>>>> 1.Created a table 'test' with 'course' in which ttl=5.
>>>>> 2. inserted one row into the table. 5 seconds later, the row was deleted.
>>>>> Later when I checked the log infor of 'test' table, I only found the
>>>>> inserted information but not deleted information.
>>>>>
>>>>> Can anyone tell me which information is written into hlog when data is
>>>>> deleted by ttl or in this situation, no information is written into
>>>>> the hlog. If there is no information of deletion in the log, how can
>>>>> we guarantee the data recovered by log are correct?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Yong
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>
>
>
> --
> Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB