Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Limits on HBase


+
William Kang 2010-09-06, 20:56
+
Jonathan Gray 2010-09-06, 23:10
+
William Kang 2010-09-07, 05:40
+
Himanshu Vashishtha 2010-09-07, 06:49
Copy link to this message
-
Re: Limits on HBase
but yes you will not be having different versions of those objects as they
are not stored as such in a table. So, that's the down side. In case your
objects are write once read multi types, I think it should work.

Let's see what others say :)

~Himanshu
On Tue, Sep 7, 2010 at 12:49 AM, Himanshu Vashishtha <[EMAIL PROTECTED]
> wrote:

> Assuming you will be using hdfs as the file system: wouldn't saving those
> large objects in the fs and keeping a pointer to them in a hbase table serve
> the purpose.
>
> [I haven't done it myself but I can't see it not working. In fact, I
> remember reading it somewhere in the list.]
>
> ~Himanshu
>
>
> On Mon, Sep 6, 2010 at 11:40 PM, William Kang <[EMAIL PROTECTED]>wrote:
>
>> Hi JG,
>> Thanks for your reply. As far as I have read in Hbase's documentation and
>> wiki, the cell size is not supposed to be larger than 10 MB. For the row,
>> I
>> am not quite sure, but it looks like 256 MB is the upper limit. I am
>> considering store some binary data used to be stored in RDBM blob field.
>> The
>> size of those binary objects may vary from hundreds of KB to hundreds of
>> MB.
>> What would be a good way to use Hbase for it? We really want to use hbase
>> to
>> avoid that scaling problem.
>> Many thanks.
>>
>>
>> William
>>
>> On Mon, Sep 6, 2010 at 7:10 PM, Jonathan Gray <[EMAIL PROTECTED]> wrote:
>>
>> > I'm not sure what you mean by "optimized cell size" or whether you're
>> just
>> > asking about practical limits?
>> >
>> > HBase is generally used with cells in the range of tens of bytes to
>> > hundreds of kilobytes.  However, I have used it with cells that are
>> several
>> > megabytes, up to about 50MB.  Up at that level, I have seen some weird
>> > performance issues.
>> >
>> > The most important thing is to be sure to tweak all of your settings.
>>  If
>> > you have 20MB cells, you need to be sure to increase the flush size
>> beyond
>> > 64MB and the split size beyond 256MB.  You also need enough memory to
>> > support all this large object allocation.
>> >
>> > And of course, test test test.  That's the easiest way to see if what
>> you
>> > want to do will work :)
>> >
>> > When you run into problems, e-mail the list.
>> >
>> > As far as row size is concerned, the only issue is that a row can never
>> > span multiple regions so a given row can only be in one region and thus
>> be
>> > hosted on one server at a time.
>> >
>> > JG
>> >
>> > > -----Original Message-----
>> > > From: William Kang [mailto:[EMAIL PROTECTED]]
>> > > Sent: Monday, September 06, 2010 1:57 PM
>> > > To: hbase-user
>> > > Subject: Limits on HBase
>> > >
>> > > Hi folks,
>> > > I know this question may have been asked many times, but I am
>> wondering
>> > > if
>> > > there is any update on the optimized cell size (in megabytes) and row
>> > > size
>> > > (in megabytes)? Many thanks.
>> > >
>> > >
>> > > William
>> >
>>
>
>
+
Andrew Purtell 2010-09-07, 18:22
+
William Kang 2010-09-08, 02:36
+
Jonathan Gray 2010-09-08, 03:30
+
William Kang 2010-10-13, 08:31
+
Sean Bigdatafun 2010-10-15, 00:23
+
Ryan Rawson 2010-10-15, 01:41
+
Sean Bigdatafun 2010-10-15, 09:20
+
William Kang 2010-09-08, 04:07
+
Ryan Rawson 2010-09-08, 04:36