Hi,

 I did a quick lookup and found these two threads related to
FuzzyRowFilter status:
https://issues.apache.org/jira/browse/HBASE-7628
https://issues.apache.org/jira/browse/HBASE-6509

   It seems that FuzzyRowFilter will be available on 96 and there is a
plan to port it to 94.5. So that is the fastest way to get this function,
say, it is NOT available in 94.2-94.4?
Best Regards,
Wei

You're right.

HBASE-7628 <https://issues.apache.org/jira/browse/HBASE-7628> would
backport this feature to 0.94.5

Cheers

On Wed, Jan 23, 2013 at 11:08 AM, Wei Tan <[EMAIL PROTECTED]> wrote:

> Hi,
>
>  I did a quick lookup and found these two threads related to
> FuzzyRowFilter status:
> https://issues.apache.org/jira/browse/HBASE-7628
> https://issues.apache.org/jira/browse/HBASE-6509
>
>    It seems that FuzzyRowFilter will be available on 96 and there is a
> plan to port it to 94.5. So that is the fastest way to get this function,
> say, it is NOT available in 94.2-94.4?
>
>
> Best Regards,
> Wei
>
>

Hi,

I am using HBASE 0.92.1+156 thought CDH 4.1.1 version.  In my test cluster with 5 region servers, disk space utilization is reached to 84.39 % ( DFS is still remaining 17,22 GB out of 469GB configured storage).

4 out 5 region servers went down today morning. Everything else is running fine. Going through the logs, I found same error message in NameServer.log, HMaster log and all 4 region server logs before they went down.  
Summary of log message is -> File /hbase/xyz could only be replicated to 0 nodes instead of minReplication (=1).  There are 5 datanode(s) running and no node(s) are excluded in this operation.

I am trying to investigate why these region servers went down when there is still space left on the disk.

Please help,
Saurabh.
Detailed error message from ->

NameNode ->
2013-01-23 06:05:37,797 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hbase (auth:SIMPLE) cause:java.io.IOException: File /hbase/elf_log/d38a7249b7be279f6e443c382f4dcd98/.tmp/1f87bc4bff3540189c83771cd5d0c4ef could only be replicated to 0 nodes instead of minReplication (=1).  There are 5 datanode(s) running and no node(s) are excluded in this operation.
2013-01-23 06:05:37,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 11 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 10.42.105.91:53474: error: java.io.IOException: File /hbase/elf_log/d38a7249b7be279f6e443c382f4dcd98/.tmp/1f87bc4bff3540189c83771cd5d0c4ef could only be replicated to 0 nodes instead of minReplication (=1).  There are 5 datanode(s) running and no node(s) are excluded in this operation.
java.io.IOException: File /hbase/elf_log/d38a7249b7be279f6e443c382f4dcd98/.tmp/1f87bc4bff3540189c83771cd5d0c4ef could only be replicated to 0 nodes instead of minReplication (=1).  There are 5 datanode(s) running and no node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:471)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
2013-01-23 06:05:38,300 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault: Not able to place enough replicas, still in need of 1 to reach 1
For more information, please enable DEBUG log level on org.apache.commons.logging.impl.Log4JLogger
2013-01-23 06:05:38,300 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* InvalidateBlocks: ask 10.42.105.91:50010 to delete [blk_-890878099230127293_174462]
2013-01-23 06:05:38,300 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* InvalidateBlocks: ask 10.42.105.112:50010 to delete [blk_7536395638665332051_174484, blk_-890878099230127293_174462, blk_-7077139403028631837_174485, blk_-1592777521358427003_174452, blk_1853843283421613770_174486, blk_-4035888711536060826_174456]
2013-01-23 06:05:38,561 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* addStoredBlock: blockMap updated: 10.42.105.112:50010 is added to blk_-7288320074167043222_174501{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[10.42.105.112:50010|RBW]]} size 67108864
2013-01-23 06:05:38,562 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault: Not able to place enough replicas, still in need of 3 to reach 3

HMaster Node ->

2013-01-23 06:05:37,962 ERROR org.apache.hadoop.hbase.master.HMaster: Region server ^@^@vm-8aa9-fe74.nam.nsroot.net,60020,1358896553048 reported a fatal error:
ABORTING region server vm-8aa9-fe74.nam.nsroot.net,60020,1358896553048: Replay of HLog required. Forcing server shutdown
Cause:
org.apache.hadoop.hbase.DroppedSnapshotException: region: elf_log,153299:1358937530530:1430447649612607:vm-cb03-2277.nam.nsroot.net:,1358938680796.d38a7249b7be279f6e443c382f4dcd98.
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1297)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1181)
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1123)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:400)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:374)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:243)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /hbase/elf_log/d38a7249b7be279f6e443c382f4dcd98/.tmp/1f87bc4bff3540189c83771cd5d0c4ef could only be replicated to 0 nodes instead of minReplication (=1).  There are 5 datanode(s) running and no node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
        at