Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Essential column family performance


+
James Taylor 2013-04-07, 06:05
+
Ted Yu 2013-04-07, 14:44
+
James Taylor 2013-04-07, 18:37
+
Ted Yu 2013-04-07, 23:03
+
Ted Yu 2013-04-07, 23:13
+
lars hofhansl 2013-04-08, 03:52
+
Ted Yu 2013-04-08, 14:49
+
Anoop John 2013-04-08, 17:10
+
James Taylor 2013-04-08, 17:38
+
Ted Yu 2013-04-08, 17:42
+
Ted Yu 2013-04-08, 18:02
+
ramkrishna vasudevan 2013-04-08, 17:51
+
Sergey Shelukhin 2013-04-08, 20:34
+
Ted Yu 2013-04-08, 21:15
+
lars hofhansl 2013-04-08, 21:41
+
James Taylor 2013-04-09, 01:53
+
lars hofhansl 2013-04-09, 23:47
+
Ted Yu 2013-04-10, 00:03
+
Ted Yu 2013-04-09, 02:51
+
Jean-Marc Spaggiari 2013-04-08, 17:19
+
Ted Yu 2013-04-08, 17:23
+
Michael Segel 2013-04-08, 18:07
Copy link to this message
-
Re: Essential column family performance
lars hofhansl 2013-04-08, 21:29
In this case it is handled all at the server, and if doing scans you still get the benefits of the sequential access pattern (rather doing a lot of seeks for point Gets).

-- Lars

________________________________
 From: Jean-Marc Spaggiari <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, April 8, 2013 10:19 AM
Subject: Re: Essential column family performance
 
Something I'm not getting, why not using separate tables instead of
CFs for a single table? Simply name your table tablename_cfname then
you get ride of the CF# limitation?

Or is there big pros to have CFs?

JM

2013/4/8 Anoop John <[EMAIL PROTECTED]>:
> Agree here. The effectiveness depends on what % of data satisfies the
> condition, how it is distributed across HFile blocks. We will get
> performance gain when the we will be able to skip some HFile blocks (from
> non essential CFs). Can test with different HFile block size (lower value)?
>
> -Anoop-
>
>
> On Mon, Apr 8, 2013 at 8:19 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> I made the following change in TestJoinedScanners.java:
>>
>> -      int flag_percent = 1;
>> +      int flag_percent = 40;
>>
>> The test took longer but still favors joined scanner.
>> I got some new results:
>>
>> 2013-04-08 07:46:06,959 INFO  [main] regionserver.TestJoinedScanners(157):
>> Slow scanner finished in 7.424388 seconds, got 2050 rows
>> ...
>> 2013-04-08 07:46:12,010 INFO  [main] regionserver.TestJoinedScanners(157):
>> Joined scanner finished in 5.05063 seconds, got 2050 rows
>>
>> 2013-04-08 07:46:18,358 INFO  [main] regionserver.TestJoinedScanners(157):
>> Slow scanner finished in 6.348517 seconds, got 2050 rows
>> ...
>> 2013-04-08 07:46:22,946 INFO  [main] regionserver.TestJoinedScanners(157):
>> Joined scanner finished in 4.587545 seconds, got 2050 rows
>>
>> Looks like effectiveness of joined scanner is affected by distribution of
>> data.
>>
>> Cheers
>>
>> On Sun, Apr 7, 2013 at 8:52 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>>
>> > Looking at the joined scanner test code, it sets it up such that 1% of
>> the
>> > rows match, which would somewhat be in line with James' results.
>> >
>> > In my own testing a while ago I found a 100% improvement with 0% match.
>> >
>> >
>> > -- Lars
>> >
>> >
>> >
>> > ________________________________
>> >  From: Ted Yu <[EMAIL PROTECTED]>
>> > To: [EMAIL PROTECTED]
>> > Sent: Sunday, April 7, 2013 4:13 PM
>> > Subject: Re: Essential column family performance
>> >
>> > I have attached 5416-TestJoinedScanners-0.94.txt to HBASE-5416 for your
>> > reference.
>> >
>> > On my MacBook, I got the following results from the test:
>> >
>> > 2013-04-07 16:08:17,474 INFO  [main]
>> regionserver.TestJoinedScanners(157):
>> > Slow scanner finished in 7.973822 seconds, got 100 rows
>> > ...
>> > 2013-04-07 16:08:17,946 INFO  [main]
>> regionserver.TestJoinedScanners(157):
>> > Joined scanner finished in 0.47235 seconds, got 100 rows
>> >
>> > Cheers
>> >
>> > On Sun, Apr 7, 2013 at 4:03 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> >
>> > > Looking at
>> > >
>> >
>> https://issues.apache.org/jira/secure/attachment/12564340/5416-0.94-v3.txt
>> ,
>> > I found that it didn't contain TestJoinedScanners which shows
>> > > difference in scanner performance:
>> > >
>> > >    LOG.info((slow ? "Slow" : "Joined") + " scanner finished in " +
>> > > Double.toString(timeSec)
>> > >
>> > >       + " seconds, got " + Long.toString(rows_count/2) + " rows");
>> > >
>> > > The test uses SingleColumnValueFilter:
>> > >
>> > >     SingleColumnValueFilter filter = new SingleColumnValueFilter(
>> > >
>> > >         cf_essential, col_name, CompareFilter.CompareOp.EQUAL,
>> flag_yes);
>> > > It is possible that the custom filter you were using would exhibit
>> > > different access pattern compared to SingleColumnValueFilter. e.g. does
>> > > your filter utilize hint ?
>> > > It would be easier for me and other people to reproduce the issue you
>> > > experienced if you put your scenario in some test similar to
+
Lars Hofhansl 2013-04-10, 01:17
+
Ted Yu 2013-04-10, 01:21
+
Ted Yu 2013-04-10, 04:03
+
lars hofhansl 2013-04-10, 04:16
+
Anoop Sam John 2013-04-10, 05:30
+
lars hofhansl 2013-04-10, 23:02
+
Stack 2013-04-10, 23:35
+
Ted Yu 2013-04-10, 23:05
+
Lars H 2013-04-10, 01:05