I have an Accumulo cluster using a single-instance (non-ensemble)
zookeeper node.  The local ZK storage was deleted, so I lost all the ZK
meta info for Accumulo.   As you can imagine, starting the master node
fails w/zk node exceptions.

Is there a way I can recover the lost Accumulo ZK state and retain my
current tables?  All HDFS data is intact.

This is not a test-cluster, so it isn't too big of a deal if I have to
re-initialize.  That said, it would still be nice to recover the ZK state.

Scott

While it is possible to go about recreating the zookeeper data, the easiest way would be to move your accumulo folder to a different name, reinitialize the instance, recreate your tables, and start bulk importing the old files. We store all of the various table settings and user permissions in zookeeper, so you lost a lot of table specific information that would be easier to recreate via internal mechanisms than manually.

If performance is a must, you can dump the !METADATA rfiles to determine split points to minimize splitting time after the import.

John

----- Original Message -----
| From: [EMAIL PROTECTED]
| To: [EMAIL PROTECTED]
| Sent: Monday, December 19, 2011 11:24:51 PM
| Subject: Recover from losing zookeeper data
| I have an Accumulo cluster using a single-instance (non-ensemble)
| zookeeper node. The local ZK storage was deleted, so I lost all the ZK
| meta info for Accumulo. As you can imagine, starting the master node
| fails w/zk node exceptions.
|
| Is there a way I can recover the lost Accumulo ZK state and retain my
| current tables? All HDFS data is intact.
|
| This is not a test-cluster, so it isn't too big of a deal if I have to
| re-initialize. That said, it would still be nice to recover the ZK
| state| .
|
| Scott

I agree with John, but one caveat to his solution is that data you have
deleted could reappear through bulk loading files that are in the HDFS
accumulo directory. This also implies that counts on aggregate tables could
be different after the operation. If your application is sensitive to these
things, then you might have to attempt to recreate the zookeeper data by
hand (or just start from scratch).

Adam
On Tue, Dec 20, 2011 at 8:10 AM, John W Vines <[EMAIL PROTECTED]> wrote:

> While it is possible to go about recreating the zookeeper data, the
> easiest way would be to move your accumulo folder to a different name,
> reinitialize the instance, recreate your tables, and start bulk importing
> the old files. We store all of the various table settings and user
> permissions in zookeeper, so you lost a lot of table specific information
> that would be easier to recreate via internal mechanisms than manually.
>
> If performance is a must, you can dump the !METADATA rfiles to determine
> split points to minimize splitting time after the import.
>
> John
>
> ----- Original Message -----
> | From: [EMAIL PROTECTED]
> | To: [EMAIL PROTECTED]
> | Sent: Monday, December 19, 2011 11:24:51 PM
> | Subject: Recover from losing zookeeper data
> | I have an Accumulo cluster using a single-instance (non-ensemble)
> | zookeeper node. The local ZK storage was deleted, so I lost all the ZK
> | meta info for Accumulo. As you can imagine, starting the master node
> | fails w/zk node exceptions.
> |
> | Is there a way I can recover the lost Accumulo ZK state and retain my
> | current tables? All HDFS data is intact.
> |
> | This is not a test-cluster, so it isn't too big of a deal if I have to
> | re-initialize. That said, it would still be nice to recover the ZK
> | state> | .
> |
> | Scott
>

On Tue, Dec 20, 2011 at 1:50 PM, Adam Fuchs <[EMAIL PROTECTED]> wrote:
> I agree with John, but one caveat to his solution is that data you have
> deleted could reappear through bulk loading files that are in the HDFS
> accumulo directory. This also implies that counts on aggregate tables could
> be different after the operation. If your application is sensitive to these
> things, then you might have to attempt to recreate the zookeeper data by
> hand (or just start from scratch).
>
> Adam
>

What John proposed is the simplest solution.  Zookeeper holds table
name to table id map, table configuration, user information, and
information about the root tablet.  The most important info that it
contains w/ regards to actual data in accumulo is the write ahead logs
for the root tablet.  Except for the root table write ahead logs, the
!METADATA table is all on hdfs.  The !METADATA table contains all the
information about which tables use which files.  This information is
important for avoiding the problem that Adam mentioned.

When you say all local ZK storage was deleted, do you mean you lost all the
data in the version-2 directory, or deleted the zk ephemera?

If it's just the latter, than you may be able to roll-back to a earlier
snapshot of your data.

Clint

On Tue, Dec 20, 2011 at 2:04 PM, Keith Turner <[EMAIL PROTECTED]> wrote:

> On Tue, Dec 20, 2011 at 1:50 PM, Adam Fuchs <[EMAIL PROTECTED]> wrote:
> > I agree with John, but one caveat to his solution is that data you have
> > deleted could reappear through bulk loading files that are in the HDFS
> > accumulo directory. This also implies that counts on aggregate tables
> could
> > be different after the operation. If your application is sensitive to
> these
> > things, then you might have to attempt to recreate the zookeeper data by
> > hand (or just start from scratch).
> >
> > Adam
> >
>
> What John proposed is the simplest solution.  Zookeeper holds table
> name to table id map, table configuration, user information, and
> information about the root tablet.  The most important info that it
> contains w/ regards to actual data in accumulo is the write ahead logs
> for the root tablet.  Except for the root table write ahead logs, the
> !METADATA table is all on hdfs.  The !METADATA table contains all the
> information about which tables use which files.  This information is
> important for avoiding the problem that Adam mentioned.
>

> When you say all local ZK storage was deleted, do you mean you lost all
> the
> data in the version-2 directory, or deleted the zk ephemera?

Lost all data.

So I ended up just re-initializing Accumulo and re-ingesting data.  Thanks
for the suggestions everyone.  The is useful troubleshooting info to have
archived.

>
> If it's just the latter, than you may be able to roll-back to a earlier
> snapshot of your data.
>
> Clint
>
> On Tue, Dec 20, 2011 at 2:04 PM, Keith Turner <[EMAIL PROTECTED]> wrote:
>
>> On Tue, Dec 20, 2011 at 1:50 PM, Adam Fuchs <[EMAIL PROTECTED]>
>> wrote:
>> > I agree with John, but one caveat to his solution is that data you
>> have
>> > deleted could reappear through bulk loading files that are in the HDFS
>> > accumulo directory. This also implies that counts on aggregate tables
>> could
>> > be different after the operation. If your application is sensitive to
>> these
>> > things, then you might have to attempt to recreate the zookeeper data
>> by
>> > hand (or just start from scratch).
>> >
>> > Adam
>> >
>>
>> What John proposed is the simplest solution.  Zookeeper holds table
>> name to table id map, table configuration, user information, and
>> information about the root tablet.  The most important info that it
>> contains w/ regards to actual data in accumulo is the write ahead logs
>> for the root tablet.  Except for the root table write ahead logs, the
>> !METADATA table is all on hdfs.  The !METADATA table contains all the
>> information about which tables use which files.  This information is
>> important for avoiding the problem that Adam mentioned.
>>
>