We were also caught by this - we're running CentOS 6. Likewise, once we reset the date/time, HBase was happy again. I wonder why Java processes seemed to be affected more than other processes?
On Jul 2, 2012, at 8:34 AM, Dean Banks wrote:
> We were caught by this issue. It impacted all of our JVM processes, including the full hbase/zookeeper/HDFS stack.
> Before we had determined root cause, we somewhat hastily bounced some of our services (ZK, HM), and had to forcefully kill a few RSes in an attempt to get things back online.
> At the end of the day (with root cause addressed), I was thrilled to see that our hbase cluster came back online, split logs for 10 minutes, then passed hbck as soon as all the regions were available. It is very impressive that hbase is capable of withstanding what amounted to a pretty nasty scenario without data loss.
> Nice work, guys.