Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Re: Get on a row with multiple columns


+
Varun Sharma 2013-02-09, 05:22
+
lars hofhansl 2013-02-09, 05:34
+
Varun Sharma 2013-02-09, 05:44
+
Ted Yu 2013-02-09, 05:55
+
Varun Sharma 2013-02-09, 06:05
+
lars hofhansl 2013-02-09, 06:33
+
Varun Sharma 2013-02-09, 06:45
+
Varun Sharma 2013-02-09, 06:57
Copy link to this message
-
Re: Get on a row with multiple columns
lars hofhansl 2013-02-09, 07:31
For 1) The Endpoint ships with HBase (in the example package, included by default).

2) exactly. Still places delete markers but you get to control this better.

In your case you want pass BulkDeleteProtocol.DeleteType.VERSION as delete type. This places an exact version delete marker for each KV encountered during the scan, and it does this efficiently region by region.
-- Lars
----- Original Message -----
From: Varun Sharma <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
Cc:
Sent: Friday, February 8, 2013 10:57 PM
Subject: Re: Get on a row with multiple columns

We are actually doing some filtering already using a coprocessor during
major compactions but we dont really know in advance what is going to be
trimmed out. We only know when an unfollow action happens.

Anyhow this BulkDelete looks promising. I have never done coprocessor
endpoints before, so can you help me with a couple of questions:
1) I am running hbase 0.94.3, do I need to do anything on the region server
side, any configuration to take advantage of this or can i simply follow
the javadoc which is really informative and use the endpoint in my client ?
2) This, as i read it will simply run a single scan (with filters etc.) and
simply place delete markers for all the entries that were found during the
scan.

Thanks
Varun

On Fri, Feb 8, 2013 at 10:45 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:

> The use case is like your twitter feed. Tweets from people u follow. When
> someone unfollows, you need to delete a bunch of his tweets from the
> following feed. So, its frequent, and we are essentially running into some
> extreme corner cases like the one above. We need high write throughput for
> this, since when someone tweets, we need to fanout the tweet to all the
> followers. We need the ability to do fast deletes (unfollow) and fast adds
> (follow) and also be able to do fast random gets - when a real user loads
> the feed. I doubt we will able to play much with the schema here since we
> need to support a bunch of use cases.
>
> @lars: It does not take 30 seconds to place 300 delete markers. It takes
> 30 seconds to first find which of those 300 pins are in the set of columns
> present - this invokes 300 gets and then place the appropriate delete
> markers. Note that we can have tens of thousands of columns in a single row
> so a single get is not cheap.
>
> If we were to just place delete markers, that is very fast. But when
> started doing that, our random read performance suffered because of too
> many delete markers. The 90th percentile on random reads shot up from 40
> milliseconds to 150 milliseconds, which is not acceptable for our usecase.
>
> Thanks
> Varun
>
>
> On Fri, Feb 8, 2013 at 10:33 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>
>> Can you organize your columns and then delete by column family?
>>
>> deleteColumn without specifying a TS is expensive, since HBase first has
>> to figure out what the latest TS is.
>>
>> Should be better in 0.94.1 or later since deletes are batched like Puts
>> (still need to retrieve the latest version, though).
>>
>> In 0.94.3 or later you can also the BulkDeleteEndPoint, which basically
>> let's specify a scan condition and then place specific delete marker for
>> all KVs encountered.
>>
>>
>> If you wanted to get really
>> fancy, you could hook up a coprocessor to the compaction process and
>> simply filter all KVs you no longer want (without ever placing any
>> delete markers).
>>
>>
>> Are you saying it takes 15 seconds to place 300 version delete markers?!
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>>  From: Varun Sharma <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Sent: Friday, February 8, 2013 10:05 PM
>> Subject: Re: Get on a row with multiple columns
>>
>> We are given a set of 300 columns to delete. I tested two cases:
>>
>> 1) deleteColumns() - with the 's'
>>
>> This function simply adds delete markers for 300 columns, in our case,
>
+
lars hofhansl 2013-02-09, 07:41
+
lars hofhansl 2013-02-09, 07:57
+
Varun Sharma 2013-02-09, 08:05
+
Varun Sharma 2013-02-09, 08:11
+
lars hofhansl 2013-02-09, 08:17
+
Varun Sharma 2013-02-09, 08:29
+
Jean-Marc Spaggiari 2013-02-09, 13:02
+
lars hofhansl 2013-02-09, 16:46
+
Varun Sharma 2013-02-10, 22:35
+
Anoop Sam John 2013-02-11, 12:50
+
Varun Sharma 2013-02-11, 15:36
+
Varun Sharma 2013-02-11, 16:44
+
Varun Sharma 2013-02-11, 16:44
+
Ted Yu 2013-02-09, 06:09
+
Varun Sharma 2013-02-09, 06:16
+
Ted 2013-02-09, 06:29
+
lars hofhansl 2013-02-09, 06:34
+
Mrudula Madiraju 2013-08-14, 03:52