-Re: Accumulo init over existing instance
Keith Turner 2013-10-09, 00:37
On Tue, Oct 8, 2013 at 7:50 PM, Terry P. <[EMAIL PROTECTED]> wrote:
> Thanks Jared.
> John, thanks for the warning! I lost a dev cluster once when we had to
> re-IP the Accumulo servers, but reverse DNS wasn't configured and I assumed
> that was why. Guess that wasn't.
> Keith, I read through ACCUMULO-1585 but it wasn't completely clear if the
> change proposed would also allow a server or servers in a cluster to have
> its IP address changed. I hope it will, as while having to re-IP a server
> or cluster is fairly rare, it certainly happens (as it did in our case).
I think moving from 1.4 to 1.5 will help. In 1.4 Accumulo has logger
servers that store write ahead logs/edit logs. Data stored on these
loggers are needed when a tablet server crashes. Accumulo stores pointers
to loggers using IP addresses. So if the IP address of the machine running
a logger changes, then Accumulo can no longer find the data need to recover
from a fault.
Starting w/ 1.5 Accumulo started storing write ahead logs in HDFS and the
pointers to these WAlogs are now hdfs paths. The IP addrs that are still
stored in 1.5 in zookeeper and the metadata table are more transient. For
example locations of tablets are stored in the metadata table using IP
addrs. If a tablet server dies and restarts w/ a different IP addr its
probably ok, because the tablet will just be reassigned to a different
tablet server. You may lose some locality because Accumulo prefers to
assign a tablet to the last place it compacted data, but things should
I have not tried changing IP addrs w/ a 1.5 instance, so I do not know if
there are other problems. But I do know that the walogs were a problem in
1.4 and that should no longer be a problem in 1.5.
> Thanks all,
> On Tue, Oct 8, 2013 at 5:14 PM, Keith Turner <[EMAIL PROTECTED]> wrote:
>> On Tue, Oct 8, 2013 at 6:07 PM, John Vines <[EMAIL PROTECTED]> wrote:
>>> Like Jared said, wiping /accumulo out of hdfs is all you need to do.
>>> But Accumulo still uses IP addresses internally, so I'm not quite
>>> certain you're going to achieve what you set out for.
>> Until 1.6.0 w/ ACCUMULO-1585
>>> On Tue, Oct 8, 2013 at 5:32 PM, Terry P. <[EMAIL PROTECTED]> wrote:
>>>> So reverse DNS wasn't working when I deployed my new cluster, thus all
>>>> my Tablet Servers were showing up in the Monitor as IP addresses (even
>>>> though all configuration files had hostnames only). Lesson learned: trust,
>>>> but verify (and ensure your hardened base servers still have nslookup
>>>> and/or dig on them).
>>>> Now that DNS is fixed, I want to wipe everything clean and re-init
>>>> Accumulo to ensure everything is legit using hostnames to ensure the
>>>> cluster is not tied to IP addresses.
>>>> I know I need to do a new 'accumulo init' -- I'll pass in the same
>>>> instance name, and my understanding is that will overwrite everything
>>>> currently in Zookeeper.
>>>> My question is: is there anything else I could/should do first to
>>>> "clean up" from this botched instance? E.g. should I delete all files in
>>>> HDFS, the write-ahead logs on the Tablet Servers, etc.? I'm running
>>>> Accumulo 1.4.2.