Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> MAX_FILESIZE and hbase.hregion.max.filesize  are both 10Gb


Copy link to this message
-
Re: MAX_FILESIZE and hbase.hregion.max.filesize are both 10Gb
"Yes it works, of course." It's not working for me ;) so was not sure.

It's normal to have regions under the half of the MAX_FILESIZE. When a
regions is more than MAX_FILESIZE it's splitted in 2. So one can be more,
and the other one can be less.

I will say, average 5GB will have been a good value, but even 3.6 is still
not so bad.

Have you pre-splitted the regions initially? Is it possible that you have
not-used pre-splitted regions?

You can you Hannibal to have a quick view of what the sizes are

JM

2013/7/28 Vladimir Rodionov <[EMAIL PROTECTED]>

> The final stats:
>
> Total HDFS size - 376GB
> #regions: 109 - avg. region size ~ 3.6GB
>
> Something is wrong here. I expected fewer regions. The regions get split
> at sizes much lower than
> hbase.hregion.max.filesize and/or  MAX_FILESIZE.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Vladimir Rodionov
> Sent: Sunday, July 28, 2013 3:35 PM
> To: [EMAIL PROTECTED]
> Subject: RE: MAX_FILESIZE and hbase.hregion.max.filesize are both 10Gb
>
> Yes it works, of course.
>
> Its in original post - ~ 10gB
>
> <property>
> <name>hbase.hregion.max.filesize</name>
>    <value>10737418240</value>
>    <source>hbase-site.xml</source>
> </property>
>
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Jean-Marc Spaggiari [[EMAIL PROTECTED]]
> Sent: Sunday, July 28, 2013 2:30 PM
> To: [EMAIL PROTECTED]
> Subject: Re: MAX_FILESIZE and hbase.hregion.max.filesize are both 10Gb
>
> Hi Vladimir,
>
> Is this link working for you? http://MASTERURL:60010/conf ? If yes, what
> do
> you have for hbase.hregion.max.filesize? To make sure the property below is
> considerered.
>
> For the table config, did you get it from the webui?
>
> JM
>
> 2013/7/28 Vladimir Rodionov <[EMAIL PROTECTED]>
>
> > but all regions keep getting split at 1Gb
> >
> > I have 71 regions and 70GB of data in 'usertable' despite the fact that:
> >
> > table config is:
> > {NAME => 'usertable', DEFERRED_LOG_FLUSH => 'true', MAX_FILESIZE =>
> > '10000000000', FAMILIES => [{NAME => 'cf', BLOOMFILTER => 'ROWCOL',
> > VERSIONS => '1', COMPRESSION => 'GZ'}]}
> >
> > and hbase-size.xml has the following config:
> >
> > <property>
> >    <name>hbase.hregion.max.filesize</name>
> >    <value>10737418240</value>
> >    <source>hbase-site.xml</source>
> > </property>
> >
> > HBase 0.94.6
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: [EMAIL PROTECTED]
> >
> > Confidentiality Notice:  The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited.  If you have received this message in error,
> please
> > immediately notify the sender and/or [EMAIL PROTECTED] and
> > delete or destroy any copy of this message and its attachments.
> >
>
> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or [EMAIL PROTECTED] and
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB