Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Limits on HBase


Copy link to this message
-
Re: Limits on HBase
Himanshu Vashishtha 2010-09-07, 06:49
Assuming you will be using hdfs as the file system: wouldn't saving those
large objects in the fs and keeping a pointer to them in a hbase table serve
the purpose.

[I haven't done it myself but I can't see it not working. In fact, I
remember reading it somewhere in the list.]

~Himanshu

On Mon, Sep 6, 2010 at 11:40 PM, William Kang <[EMAIL PROTECTED]>wrote:

> Hi JG,
> Thanks for your reply. As far as I have read in Hbase's documentation and
> wiki, the cell size is not supposed to be larger than 10 MB. For the row, I
> am not quite sure, but it looks like 256 MB is the upper limit. I am
> considering store some binary data used to be stored in RDBM blob field.
> The
> size of those binary objects may vary from hundreds of KB to hundreds of
> MB.
> What would be a good way to use Hbase for it? We really want to use hbase
> to
> avoid that scaling problem.
> Many thanks.
>
>
> William
>
> On Mon, Sep 6, 2010 at 7:10 PM, Jonathan Gray <[EMAIL PROTECTED]> wrote:
>
> > I'm not sure what you mean by "optimized cell size" or whether you're
> just
> > asking about practical limits?
> >
> > HBase is generally used with cells in the range of tens of bytes to
> > hundreds of kilobytes.  However, I have used it with cells that are
> several
> > megabytes, up to about 50MB.  Up at that level, I have seen some weird
> > performance issues.
> >
> > The most important thing is to be sure to tweak all of your settings.  If
> > you have 20MB cells, you need to be sure to increase the flush size
> beyond
> > 64MB and the split size beyond 256MB.  You also need enough memory to
> > support all this large object allocation.
> >
> > And of course, test test test.  That's the easiest way to see if what you
> > want to do will work :)
> >
> > When you run into problems, e-mail the list.
> >
> > As far as row size is concerned, the only issue is that a row can never
> > span multiple regions so a given row can only be in one region and thus
> be
> > hosted on one server at a time.
> >
> > JG
> >
> > > -----Original Message-----
> > > From: William Kang [mailto:[EMAIL PROTECTED]]
> > > Sent: Monday, September 06, 2010 1:57 PM
> > > To: hbase-user
> > > Subject: Limits on HBase
> > >
> > > Hi folks,
> > > I know this question may have been asked many times, but I am wondering
> > > if
> > > there is any update on the optimized cell size (in megabytes) and row
> > > size
> > > (in megabytes)? Many thanks.
> > >
> > >
> > > William
> >
>