Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - MemStoreFlusher: region has too many store files (client timeout)


+
kzurek 2013-02-04, 12:45
Copy link to this message
-
Re: MemStoreFlusher: region has too many store files (client timeout)
Kevin O'dell 2013-02-04, 14:46
Kzurek,

Just because you turn off time based major compactions, it does not mean
that you have turned major compaction off.  Compaction can still be
promoted to be Majors.  Also, the only real difference between a major and
minor compaction is one processes deletes.  You should really schedule at
least daily major compactions.  As for your blocking issue, there are quite
a few things you would want to look at:

How often are you flushing?

How often are you force flushing from HLog rolls?

What size are your flushes?

What does your region count look like as that can affect your flush size?

etc

When I see HBase blocking constantly it is usually a sign that you need to
do some fine grain tuning.

On Mon, Feb 4, 2013 at 7:45 AM, kzurek <[EMAIL PROTECTED]> wrote:

> I'm facing some issues regarding to major compaction. I've disabled major
> compaction and it is not triggered manually, but when I'm loading data to
> selected region, I saw that major compaction queue is growing and it is
> being triggered ('Large Compaction' in logs) quite often (mainly due to
> cacheFlusher). Moreover, I've noticed that my clients app gets timeout
> while
> putting data into the cluster (happens when memory store flusher is trying
> to dump memory content to store file, but it cannot due to too many store
> files), also drop in data rate, which in this case is obvious, is
> noticeable.  For me, it looks like compaction process is not fast enough
> comparing to incoming rate of data or ... maybe something else?? and
> finally
> is blocking the update process.
>
> Basic information:
> HBase Version: 0.92.1, r1298924
> Hadoop Version: 1.0.3, r1335192
>
> Logs:
> 2013-02-01 15:43:14,627 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Large Compaction
> requested:
>
> regionName=test,\x00\x00\x00\x00~i\x91\x00\x00\x00\x0D,1359115210217.3b710693d6314c2a987b07dd82451158.,
> storeName=data, fileCount=3, fileSize=478.3m (249.8m, 113.7m, 114.7m),
> priority=-3, time=1051078047102762; Because:
> regionserver60020.cacheFlusher;
> compaction_queue=(48:0), split_queue=0
> 2013-02-01 15:43:14,627 DEBUG org.apache.hadoop.hbase.regionserver.Store:
> 3b710693d6314c2a987b07dd82451158 - tag: no store files to compact
> 2013-02-01 15:43:14,709 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
>
> {"processingtimems":61908,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@57a56081
> ),
> rpc version=1, client version=29,
> methodsFingerPrint=54742778","client":"192.168.1.68:49419
> ","starttimems":1359729732799,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
> 2013-02-01 15:43:14,710 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> Responder, call multi(org.apache.hadoop.hbase.client.MultiAction@57a56081
> ),
> rpc version=1, client version=29, methodsFingerPrint=54742778 from
> 192.168.1.68:49419: output error
> 2013-02-01 15:43:14,710 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60020 caught: java.nio.channels.ClosedChannelException
>         at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
>         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer.channelIO(HBaseServer.java:1710)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
>         at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
>         at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
>         at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
>
> 2013-02-01 15:43:19,397 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> Flush requested on
>
> test,\x00\x00\x00\x00~i\x91\x00\x00\x00\x0D,1359115210217.3b710693d6314c2a987b07dd82451158.
> 2013-02-01 15:43:19,397 WARN

Kevin O'Dell
Customer Operations Engineer, Cloudera
+
kzurek 2013-02-05, 08:39
+
Kevin Odell 2013-02-05, 14:20
+
kzurek 2013-02-05, 14:52