Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Understanding HBase log output regarding memstore flush


Copy link to this message
-
Understanding HBase log output regarding memstore flush
Hi,

I'm trying to figure out some discrepancies I'm witnessing in the HBase
Region Server log file.

It states that a flush was requested, and then a memstore flush is started.
It says the flush size, after snapshotting is 139105600 (~132.7m).
In the log message below, the file size of the file the memstore was
flushed too is not the same size (132.7m):

[Quote]
Added
hdfs://dror.foo.local:8020/hbase/food_logs/64fd0d8da5714f03eb67f7f788a99960/data/8aa2974bdd7c4222a783b9b1558f9915,
entries=214652, sequenceid=1068343, filesize=*35.5m*
[/Quote]

I'm curios to this difference in size, since I eventually would like to
know the HDFS write throughput HBase is experiencing while flushing
memstore to disk.
*Logs quote*

2012-06-25 16:23:11,905 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Flush requested on
food_logs,,1340630544300.64fd0d8da5714f03eb67f7f788a99960.
2012-06-25 16:23:11,906 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
food_logs,,1340630544300.64fd0d8da5714f03eb67f7f788a99960., current region
memstore size 132.7m
2012-06-25 16:23:11,906 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
food_logs,,1340630544300.64fd0d8da5714f03eb67f7f788a99960., commencing wait
for mvcc, flushsize=139105600
2012-06-25 16:23:11,906 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2012-06-25 16:23:11,938 DEBUG org.apache.hadoop.hbase.util.FSUtils:
Creating
file:hdfs://dror.foo.local:8020/hbase/food_logs/64fd0d8da5714f03eb67f7f788a99960/.tmp/8aa2974bdd7c4222a783b9b1558f9915with
permission:rwxrwxrwx
2012-06-25 16:23:11,960 DEBUG
org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with
CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
[cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
[cacheEvictOnClose=false] [cacheCompressed=false]
2012-06-25 16:23:11,960 INFO
org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter
type for
hdfs://dror.foo.local:8020/hbase/food_logs/64fd0d8da5714f03eb67f7f788a99960/.tmp/8aa2974bdd7c4222a783b9b1558f9915:
 CompoundBloomFilterWriter
2012-06-25 16:23:12,218 DEBUG
org.apache.hadoop.hbase.regionserver.LogRoller: HLog roll requested
2012-06-25 16:23:13,043 INFO
org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO
DeleteFamily was added to HFile (
hdfs://dror.foo.local:8020/hbase/food_logs/64fd0d8da5714f03eb67f7f788a99960/.tmp/8aa2974bdd7c4222a783b9b1558f9915
)
2012-06-25 16:23:13,043 INFO org.apache.hadoop.hbase.regionserver.Store:
Flushed , sequenceid=1068343, memsize=132.7m, into tmp file
hdfs://dror.foo.local:8020/hbase/food_logs/64fd0d8da5714f03eb67f7f788a99960/.tmp/8aa2974bdd7c4222a783b9b1558f9915
2012-06-25 16:23:13,050 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Renaming flushed file at
hdfs://dror.foo.local:8020/hbase/food_logs/64fd0d8da5714f03eb67f7f788a99960/.tmp/8aa2974bdd7c4222a783b9b1558f9915
 to
hdfs://dror.foo.local:8020/hbase/food_logs/64fd0d8da5714f03eb67f7f788a99960/data/8aa2974bdd7c4222a783b9b1558f9915
2012-06-25 16:23:13,068 INFO org.apache.hadoop.hbase.regionserver.Store:
Added
hdfs://dror.foo.local:8020/hbase/food_logs/64fd0d8da5714f03eb67f7f788a99960/data/8aa2974bdd7c4222a783b9b1558f9915,
entries=214652, sequenceid=1068343, filesize=35.5m
2012-06-25 16:23:13,071 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~132.7m/139105600, currentsize=9.2m/9603424 for
region food_logs,,1340630544300.64fd0d8da5714f03eb67f7f788a99960. in
1165ms, sequenceid=1068343, compaction requested=true
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB