Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - RegionServer crash without any errors (compaction?)


Copy link to this message
-
Re: RegionServer crash without any errors (compaction?)
John 2013-11-07, 16:09
there are no really other logs before. There are a "operationTooSlow"
message before, but that log is ~50 mins bofre the other:
http://pastebin.com/EAAubqGB
2013/11/7 John <[EMAIL PROTECTED]>

> Hi,
>
> thanks for your fast answer. If I take a look at the cloudera manager at
> this time the %-time of using the GC increase at this time, so I think you
> are right. The max heap size is 1GB for this node. The
> hbase.hregion.max.filesize is also 1GB.
>
> regards
>
>
> 2013/11/7 Dhaval Shah <[EMAIL PROTECTED]>
>
>> Did you look at your GC logs? Probably the compaction process is running
>> your region server out of memory. Can you provide more details on your
>> setup? Max heap size? Max Region HFile size?
>>
>> Regards,
>> Dhaval
>>
>>
>> ________________________________
>>  From: John <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Sent: Thursday, 7 November 2013 10:51 AM
>> Subject: RegionServer crash without any errors (compaction?)
>>
>>
>> Hi,
>>
>> I have a cluster with 7 regionserver. Some of them are crashing from time
>> to time wihtout any error message in the hbase log. If I take a look at
>> the
>> log at the time I found this:
>>
>> 2013-11-07 15:29:02,511 INFO org.apache.hadoop.hbase.regionserver.Store:
>> Starting compaction of 2 file(s) in 1 of P_SO,<
>> http://xmlns.com/foaf/0.1/homepage
>> >,1383188177383.59d0259c87c07dc666a5600ba4d6c916.
>> i$
>> 2013-11-07 15:29:10,471 INFO
>> org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter
>> type for hdfs://
>>
>> pc08.pool.ifis.uni-luebeck.de:8020/hbase/P_SO/59d0259c87c07dc666a5600ba4d6c916/.tmp/f$
>> 2013-11-07<http://pc08.pool.ifis.uni-luebeck.de:8020/hbase/P_SO/59d0259c87c07dc666a5600ba4d6c916/.tmp/f$2013-11-07>15:31:05,944 INFO org.apache.hadoop.hbase.util.VersionInfo:
>> HBase 0.94.6-cdh4.4.0
>> .... restart
>>
>> At this time 2 of the 7 RS crashed, both has this compaction message
>> before
>> they crashed. I don't know exactly what compaction is, but it seems that
>> this compaction has to do with the crash. What can I do to avoid this
>> restart/crash?
>>
>> best regards
>>
>
>