El 01/10/2012 16:35, Greg Ross escribió:
> Hi,
>
> I'm having difficulty with a mapreduce job that has reducers that read
> from and write to HBase, version 0.92.1, r1298924. Row sizes vary
> greatly. As do the number of cells, although the number of cells is
> typically numbered in the tens, at most. The max cell size is 1MB.
0.94.1 is out with a lot of improvements related to performance. It
would better
if you use tis version.
>
> I see the following in the logs followed by the region server promptly
> shutting down:
>
> 2012-10-01 19:08:47,858 [regionserver60020] WARN
> org.apache.hadoop.hbase.util.Sleeper: We slept 28970ms instead of
> 3000ms, this is likely due to a long garbage collecting pause and it's
> usually bad, see
> http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
>
> The full logs, including GC are below.
>
> Although new to HBase, I've read up on the likely GC issues and their
> remedies. I've implemented the recommended solutions and still to no
> avail.
>
> Here's what I've tried:
>
> (1) increased the RAM to 4G
Which is the exact size of your RAM?
> (2) set -XX:+UseConcMarkSweepGC
> (3) set -XX:+UseParNewGC
> (4) set -XX:CMSInitiatingOccupancyFraction=N where I've attempted N=[40..70]
> (5) I've called context.progress() in the reducer before and after
> reading and writing
> (6) memstore is enabled
>
> Is there anything else that I might have missed?
>
> Thanks,
>
> Greg
I´m seeing in the HBase logs that there a lot of block requests that are
failing.
Can you send to us the ouput of JPS?
Can you check the filesystem´s health with hadoop fsck?

Another thing that I´m seeing is that one of your main process is
compaction,
so you can optimize all this inceasing the size of your regions (by
defaulf the size of a
region is 256 MB), but you will have in your hands a "split/compaction
storm" like
Lars called them on his book.

Instead using the default mechanism for region spliting and compaction,
you can turn it
off and do it manually with the split and major_compaction commands.

You can evaluate to use compresion in your cluster to save a lot of
space in your region servers.

Which is the size of your cluster?
You can use SPM, Ganglia or OpenTSDB to monitor constantly your cluster.

Best wishes
>
> hbase logs
> =======>
> 2012-10-01 19:09:48,293
> [regionserver60020-largeCompactions-1348577979539] INFO
> org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at
> hdfs://namenode301.ngpipes.milp.ngmoco.com:9000/hbase/orwell_events/a9906c96a91bb8d7e62a7a528bf0ea5c/.tmp/d2ee47650b224189b0c27d1c20929c03
> to hdfs://namenode301.ngpipes.milp.ngmoco.com:9000/hbase/orwell_events/a9906c96a91bb8d7e62a7a528bf0ea5c/U/d2ee47650b224189b0c27d1c20929c03
> 2012-10-01 19:09:48,884
> [regionserver60020-largeCompactions-1348577979539] INFO
> org.apache.hadoop.hbase.regionserver.Store: Completed major compaction
> of 5 file(s) in U of
> orwell_events,00321084,1349118541283.a9906c96a91bb8d7e62a7a528bf0ea5c.
> into d2ee47650b224189b0c27d1c20929c03, size=723.0m; total size for
> store is 723.0m
> 2012-10-01 19:09:48,884
> [regionserver60020-largeCompactions-1348577979539] INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest:
> completed compaction:
> regionName=orwell_events,00321084,1349118541283.a9906c96a91bb8d7e62a7a528bf0ea5c.,
> storeName=U, fileCount=5, fileSize=1.4g, priority=2,
> time=10631266687564968; duration=35sec
> 2012-10-01 19:09:48,886
> [regionserver60020-largeCompactions-1348577979539] INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on U
> in region orwell_events,00316914,1349118541283.9740f22a42e9e8b6aca3966c0173e680.
> 2012-10-01 19:09:48,887
> [regionserver60020-largeCompactions-1348577979539] INFO
> org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 5
> file(s) in U of
> orwell_events,00316914,1349118541283.9740f22a42e9e8b6aca3966c0173e680.
> into tmpdir=hdfs://namenode301.ngpipes.milp.ngmoco.com:9000/hbase/orwell_events/9740f22a42e9e8b6aca3966c0173e680/.tmp,
Marcos Ortiz Valmaseda,
Data Engineer && Senior System Administrator at UCI
Blog: http://marcosluis2186.posterous.com
Linkedin: http://www.linkedin.com/in/marcosluis2186
Twitter: @marcosluis2186

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

Hello

2012/10/2 Marcos Ortiz <[EMAIL PROTECTED]>

>
> Another thing that I´m seeing is that one of your main process is
> compaction,
> so you can optimize all this inceasing the size of your regions (by
> defaulf the size of a
> region is 256 MB), but you will have in your hands a "split/compaction
> storm" like
> Lars called them on his book.
Actually it seams like the default value for hbase.hregion.max.filesize in
0.92 was increased up to 1Go.
http://hbase.apache.org/book/upgrade0.92.html#d2051e266

But you can set it to more (max is 20Go) and split manually.
http://hbase.apache.org/book/important_configurations.html#bigger.regions

Cheers,

--
Dam

Thanks for the suggestions.

I was attempting to tune the GC via mapred.child.java.opts in the job's
Oozie config instead of in hbase-env.sh. I think this is why my efforts
were to no avail. It was likely having no effect on the read/write
performance. Is there any way of specifying job-specific HBase parameters
instead of globally setting them in hbase-env.sh?

The cluster has 175 nodes. Each with 48GB of RAM. The overall data input
size is 7TB and I pre-split the table into initially 30 regions, then 100
in another attempt. Each job runs upon 700GB chunks of the data. I used
RegionSplitter to create and condition the table and therefore there's
currently no compression. I'm thinking to recreate the table and 'alter' it
with LZO compression before attempting the jobs again.

Cheers.

Greg

On Tue, Oct 2, 2012 at 7:20 AM, Damien Hardy <[EMAIL PROTECTED]> wrote:

> Hello
>
> 2012/10/2 Marcos Ortiz <[EMAIL PROTECTED]>
>
>>
>> Another thing that I´m seeing is that one of your main process is
>> compaction,
>> so you can optimize all this inceasing the size of your regions (by
>> defaulf the size of a
>> region is 256 MB), but you will have in your hands a "split/compaction
>> storm" like
>> Lars called them on his book.
>
>
> Actually it seams like the default value for hbase.hregion.max.filesize in
> 0.92 was increased up to 1Go.
> http://hbase.apache.org/book/upgrade0.92.html#d2051e266
>
> But you can set it to more (max is 20Go) and split manually.
> http://hbase.apache.org/book/important_configurations.html#bigger.regions
>
> Cheers,
>
> --
> Dam
>

--
*gregross:)*

El 02/10/2012 11:32, Greg Ross escribi�:
> Thanks for the suggestions.
>
> I was attempting to tune the GC via mapred.child.java.opts in the job's
> Oozie config instead of in hbase-env.sh. I think this is why my efforts
> were to no avail. It was likely having no effect on the read/write
> performance. Is there any way of specifying job-specific HBase parameters
> instead of globally setting them in hbase-env.sh?
>
> The cluster has 175 nodes. Each with 48GB of RAM. The overall data input
> size is 7TB and I pre-split the table into initially 30 regions, then 100
> in another attempt. Each job runs upon 700GB chunks of the data. I used
> RegionSplitter to create and condition the table and therefore there's
> currently no compression. I'm thinking to recreate the table and 'alter' it
> with LZO compression before attempting the jobs again.
There are many points that you can do for HBase performance tuning.
In the Lars George�s book "HBase: The Definitive Guide", the Chapter 11
is dedicated
to this tricky topic, and in the HBase book, there are good points too:

http://hbase.apache.org/book.html#perf.reading

Thanks to Doug for the link.
>
> Cheers.
>
> Greg
>
>
>
> On Tue, Oct 2, 2012 at 7:20 AM, Damien Hardy <[EMAIL PROTECTED]> wrote:
>
>> Hello
>>
>> 2012/10/2 Marcos Ortiz <[EMAIL PROTECTED]>
>>
>>> Another thing that I�m seeing is that one of your main process is
>>> compaction,
>>> so you can optimize all this inceasing the size of your regions (by
>>> defaulf the size of a
>>> region is 256 MB), but you will have in your hands a "split/compaction
>>> storm" like
>>> Lars called them on his book.
>>
>> Actually it seams like the default value for hbase.hregion.max.filesize in
>> 0.92 was increased up to 1Go.
>> http://hbase.apache.org/book/upgrade0.92.html#d2051e266
>>
>> But you can set it to more (max is 20Go) and split manually.
>> http://hbase.apache.org/book/important_configurations.html#bigger.regions
>>
>> Cheers,
>>
>> --
>> Dam
>>
>
>

--
Marcos Ortiz Valmaseda,
Data Engineer && Senior System Administrator at UCI
Blog: http://marcosluis2186.posterous.com
Linkedin: http://www.linkedin.com/in/marcosluis2186
Twitter: @marcosluis2186
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

You really don't want to go to 20GB.

Without knowing the number of regions... going beyond 1-2 GB may cause more headaches than its worth.

Sorry, but I tend to be very cautious when it comes to tuning.

-Mike

On Oct 2, 2012, at 9:20 AM, Damien Hardy <[EMAIL PROTECTED]> wrote:

> Hello
>
> 2012/10/2 Marcos Ortiz <[EMAIL PROTECTED]>
>
>>
>> Another thing that I´m seeing is that one of your main process is
>> compaction,
>> so you can optimize all this inceasing the size of your regions (by
>> defaulf the size of a
>> region is 256 MB), but you will have in your hands a "split/compaction
>> storm" like
>> Lars called them on his book.
>
>
> Actually it seams like the default value for hbase.hregion.max.filesize in
> 0.92 was increased up to 1Go.
> http://hbase.apache.org/book/upgrade0.92.html#d2051e266
>
> But you can set it to more (max is 20Go) and split manually.
> http://hbase.apache.org/book/important_configurations.html#bigger.regions
>
> Cheers,
>
> --
> Dam