Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - one column family but lots of tables


+
Koert Kuipers 2013-08-22, 15:06
+
Shahab Yunus 2013-08-22, 16:01
+
Vladimir Rodionov 2013-08-22, 17:58
+
Koert Kuipers 2013-08-22, 19:30
+
Vladimir Rodionov 2013-08-22, 21:16
+
Koert Kuipers 2013-08-22, 22:43
+
lars hofhansl 2013-08-23, 05:35
+
Michael Segel 2013-08-23, 14:33
+
Koert Kuipers 2013-08-24, 15:47
Copy link to this message
-
Re: one column family but lots of tables
yonghu 2013-08-24, 16:30
I think you can take a look at the
http://hbase.apache.org/book/regions.arch.html, it describes the data
storage hierarchy of HBase.  Due to the statement of Lars "stems from the
fact that HBase flushes by region (which means all stores of that region
are flushed)", you can think the limitations of column-families like this
way. If the size of the region is fixed, the more column-families you
create, the less space you can get for each column-family. As the flush
operation is based on region, if one column-family research its size
threshold even if the other column-families are empty, the flush operations
will be triggered. Roughly speaking, the more column-families you create,
the more I/O cost you will get. I think if the flush is based on
column-family rather than region, maybe we can release this limitation.
On Sat, Aug 24, 2013 at 5:47 PM, Koert Kuipers <[EMAIL PROTECTED]> wrote:

> thanks i think it makes sense. i will go through hbase architecture again
> to make sure i fully understand the mapping to "regions" and "stores"
>
>
> On Fri, Aug 23, 2013 at 10:33 AM, Michael Segel
> <[EMAIL PROTECTED]>wrote:
>
> > I think the issue which a lot of people miss is why do you want to use a
> > column family in the first place.
> >
> > Column families are part of the same table structure, and each family is
> > kept separate.
> >
> > So in your design, do you have tables which are related, but are not
> > always used at the same time?
> >
> > The example that I use when I teach about HBase or do a
> > lecture/presentation is an Order Entry system.
> > Here you have an order being entered, then you have  one or many pick
> > slips being generated, same for shipping then there's the invoicing.
> > All separate processes which relate back to the same order.
> >
> > So here it makes sense to use column families.
> >
> > Other areas could be metadata is surrounding a transaction. Again... few
> > column families are tied together.
> >
> > Does this make sense?
> >
> >
> > On Aug 23, 2013, at 12:35 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> >
> > > You can think of it this way: Every region and column family is a
> > "store" in HBase. Each store has a memstore and its own set of HFiles in
> > HDFS.
> > > The more stores you have, the more there is to manage.
> > >
> > > So you want to limit the number of stores. Also note that the word
> > "Table" is somewhat a misnomer in HBase it should have better been called
> > "Keyspace".
> > > The extra caution for the number of column families per table stems
> from
> > the fact that HBase flushes by region (which means all stores of that
> > region are flushed). This in turn means that unless are column families
> > hold roughly the same amount of data you end up with very lopsided
> > distributions of HFile sizes.
> > >
> > > -- Lars
> > >
> > >
> > >
> > > ________________________________
> > > From: Koert Kuipers <[EMAIL PROTECTED]>
> > > To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> > > Sent: Thursday, August 22, 2013 12:30 PM
> > > Subject: Re: one column family but lots of tables
> > >
> > >
> > > if that is the case, how come people keep warning about limiting the
> > number
> > > of column families to only a handful (with more hbase performance will
> > > degrade supposedly), yet there seems to be no similar warnings for
> number
> > > of tables? see for example here:
> > > http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/27616
> > >
> > > if a table means at least one column family then the number of tables
> > > should also be kept to a minumum, no?
> > >
> > >
> > >
> > >
> > > On Thu, Aug 22, 2013 at 1:58 PM, Vladimir Rodionov
> > > <[EMAIL PROTECTED]>wrote:
> > >
> > >> Nope. Column family is per table (its sub-directory inside your table
> > >> directory in HDFS).
> > >> If you have N tables you will always have , at least, N distinct CFs
> > (even
> > >> if they have the same name).
> > >>
> > >> Best regards,
> > >> Vladimir Rodionov
> > >> Principal Platform Engineer
+
Ted Yu 2013-08-22, 16:06
+
Koert Kuipers 2013-08-22, 16:31