Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Too many "Compaction Complete" messages


+
Tom Brown 2013-09-04, 02:13
+
Frank Chow 2013-09-04, 12:21
+
Tom Brown 2013-09-04, 15:36
+
Jimmy Xiang 2013-09-04, 15:54
+
Tom Brown 2013-09-04, 17:27
+
Tom Brown 2013-09-21, 15:49
Copy link to this message
-
Re: Too many "Compaction Complete" messages
Tom Brown 2013-09-25, 15:08
To update this thread, this was caused by a bug: HBASE-9648.

--Tom
On Sat, Sep 21, 2013 at 9:49 AM, Tom Brown <[EMAIL PROTECTED]> wrote:

> I am still receiving thousands these log messages for the same region
> withing a very short time frame. I have read the compaction documentation,
> but have not been able to reconcile the documentation with the behavior I'm
> seeing, so I'm starting to think it's a bug.
>
> To help diagnose further, I have included a recent example, hot off the
> server (see below)
>
> Thanks,
> Tom Brown
>
> This particular region (c5f15027ae1d4aa1d5b6046aea6f63a4) is about 800MB,
> comprised of 25 store files. Given that, I could reasonably expect up to 25
> messages for the region. However, there were at least 388 (I didn't look
> past the 60 second cutoff in the region server status page. However, after
> finishing this email, there were over 300 new entries. I can only assume it
> continued full-throttle during this time).
>
> Here is a small snippet from the region server status page. You'll notice
> they're all identical except for the timestamp:
>
> Sat Sep 21 09:19:10 MDT 2013 Compacting d in
> compound4,pub-aid\x09CBD46B77\x09\x80\x00\x00\x00\x00\x05\xBB/\x80\x00\x01>\xDD1\xAB`,1371513244597.c5f15027ae1d4aa1d5b6046aea6f63a4. COMPLETE
> (since 0sec ago) Compaction complete (since 0sec ago)
> Sat Sep 21 09:19:10 MDT 2013 Compacting d in
> compound4,pub-aid\x09CBD46B77\x09\x80\x00\x00\x00\x00\x05\xBB/\x80\x00\x01>\xDD1\xAB`,1371513244597.c5f15027ae1d4aa1d5b6046aea6f63a4. COMPLETE
> (since 0sec ago) Compaction complete (since 0sec ago)
> Sat Sep 21 09:19:10 MDT 2013 Compacting d in
> compound4,pub-aid\x09CBD46B77\x09\x80\x00\x00\x00\x00\x05\xBB/\x80\x00\x01>\xDD1\xAB`,1371513244597.c5f15027ae1d4aa1d5b6046aea6f63a4. COMPLETE
> (since 0sec ago) Compaction complete (since 0sec ago)
> Sat Sep 21 09:19:10 MDT 2013 Compacting d in
> compound4,pub-aid\x09CBD46B77\x09\x80\x00\x00\x00\x00\x05\xBB/\x80\x00\x01>\xDD1\xAB`,1371513244597.c5f15027ae1d4aa1d5b6046aea6f63a4. COMPLETE
> (since 0sec ago) Compaction complete (since 0sec ago)
>
>
> Here is a snippet of the region server log during this time. For each item
> in the region server status, there are 13 lines in the log file (claiming
> to have created a new store file?).
>
> 2013-09-21 09:19:10,717 INFO
> org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO
> DeleteFamily was added to HFile (hdfs://
> hdpmgr001.pse.movenetworks.com:8020/hbase/compound4/c5f15027ae1d4aa1d5b6046aea6f63a4/.tmp/885989d2a3be4a2a8d1e2df86b023f09
> )
> 2013-09-21 09:19:10,746 INFO
> org.apache.hadoop.hbase.regionserver.StoreFile: HFile Bloom filter type for
> 885989d2a3be4a2a8d1e2df86b023f09: NONE, but ROW specified in column family
> configuration
> 2013-09-21 09:19:10,746 INFO org.apache.hadoop.hbase.regionserver.Store:
> Renaming compacted file at hdfs://
> hdpmgr001.pse.movenetworks.com:8020/hbase/compound4/c5f15027ae1d4aa1d5b6046aea6f63a4/.tmp/885989d2a3be4a2a8d1e2df86b023f09to hdfs://
> hdpmgr001.pse.movenetworks.com:8020/hbase/compound4/c5f15027ae1d4aa1d5b6046aea6f63a4/d/885989d2a3be4a2a8d1e2df86b023f09
> 2013-09-21 09:19:10,784 INFO
> org.apache.hadoop.hbase.regionserver.StoreFile: HFile Bloom filter type for
> 885989d2a3be4a2a8d1e2df86b023f09: NONE, but ROW specified in column family
> configuration
> 2013-09-21 09:19:10,803 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed compaction of 1 file(s) in d of
> compound4,pub-aid\x09CBD46B77\x09\x80\x00\x00\x00\x00\x05\xBB/\x80\x00\x01>\xDD1\xAB`,1371513244597.c5f15027ae1d4aa1d5b6046aea6f63a4.
> into 885989d2a3be4a2a8d1e2df86b023f09, size=491.0; total size for store is
> 815.0m
> 2013-09-21 09:19:10,804 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest:
> completed compaction:
> regionName=compound4,pub-aid\x09CBD46B77\x09\x80\x00\x00\x00\x00\x05\xBB/\x80\x00\x01>\xDD1\xAB`,1371513244597.c5f15027ae1d4aa1d5b6046aea6f63a4.,
> storeName=d, fileCount=1, fileSize=491.0, priority=-18,