During an upgrade of my cluster to 0.90 to 0.92 over the weekend, the WAL (files in the /hbase/.logs directory) was corrupted and it prevented HBase from starting up. The exact exception was "java.io.IOException: Could not obtain the last block locations" on the WAL files.

I was able to recover by deleting the /hbase/.logs directory. My question is, if HBase had no pending updates, i.e. nothing writing to it, is there any risk of data loss by deleting the WAL directory? For example, does rebalancing, flushing, or compaction use the WAL or is the WAL used only for inserts/updates/deletes?

During an upgrade of my cluster to 0.90 to 0.92 over the weekend, the WAL (files in the /hbase/.logs directory) was corrupted and it prevented HBase from starting up. The exact exception was "java.io.IOException: Could not obtain the last block locations" on the WAL files.

I was able to recover by deleting the /hbase/.logs directory. My question is, if HBase had no pending updates, i.e. nothing writing to it, is there any risk of data loss by deleting the WAL directory? For example, does rebalancing, flushing, or compaction use the WAL or is the WAL used only for inserts/updates/deletes?

my understanding is that the WAL log is used for replication as well.
If all your data has been persisted to disk (i.e. all data in memstores have been flushed to disks) and replication is disabled, I believe you can delete the WAL without data loss.

just my 2 cents

On 2012-07-02, at 1:37 PM, Bryan Keller wrote:

> During an upgrade of my cluster to 0.90 to 0.92 over the weekend, the WAL (files in the /hbase/.logs directory) was corrupted and it prevented HBase from starting up. The exact exception was "java.io.IOException: Could not obtain the last block locations" on the WAL files.
>
> I was able to recover by deleting the /hbase/.logs directory. My question is, if HBase had no pending updates, i.e. nothing writing to it, is there any risk of data loss by deleting the WAL directory? For example, does rebalancing, flushing, or compaction use the WAL or is the WAL used only for inserts/updates/deletes?

On Mon, Jul 2, 2012 at 7:37 PM, Bryan Keller <[EMAIL PROTECTED]> wrote:
> During an upgrade of my cluster to 0.90 to 0.92 over the weekend, the WAL (files in the /hbase/.logs directory) was corrupted and it prevented HBase from starting up. The exact exception was "java.io.IOException: Could not obtain the last block locations" on the WAL files.
>

Next time, run hdfs fsck and see what it says about your WALs.

> I was able to recover by deleting the /hbase/.logs directory. My question is, if HBase had no pending updates, i.e. nothing writing to it, is there any risk of data loss by deleting the WAL directory? For example, does rebalancing, flushing, or compaction use the WAL or is the WAL used only for inserts/updates/deletes?

Only by inserts, etc.

You could have lost data if edits were in memory on regionserver only
and they had not yet been flushed.

St.Ack

Hi bryan,

WAL is used only during minor compaction. During major compaction WAL is
not used.

Cheers,
Manoj.P

On Mon, Jul 2, 2012 at 11:00 PM, Bryan Keller <[EMAIL PROTECTED]> wrote:

> During an upgrade of my cluster to 0.90 to 0.92 over the weekend, the WAL
> (files in the /hbase/.logs directory) was corrupted and it prevented HBase
> from starting up. The exact exception was "java.io.IOException: Could not
> obtain the last block locations" on the WAL files.
>
> I was able to recover by deleting the /hbase/.logs directory. My question
> is, if HBase had no pending updates, i.e. nothing writing to it, is there
> any risk of data loss by deleting the WAL directory? For example, does
> rebalancing, flushing, or compaction use the WAL or is the WAL used only
> for inserts/updates/deletes?
>
>