Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot


Copy link to this message
-
Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
lars hofhansl 2013-04-16, 17:12
Please see my last comment on the jira. We can make this work without breaking users who are using HDFS snapshots.

________________________________
 From: Ted Yu <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
Sent: Tuesday, April 16, 2013 10:00 AM
Subject: Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
 
Let's get proper release notes for HBASE-8352 .

Either Lars or I can send out notification to user mailing list so that there is enough preparation for this change.

Cheers
On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <[EMAIL PROTECTED]> wrote:

I was away from keyboard when I asserted that hdfs snapshot was a hadoop
>2.1 or 3.0 feature.  Apparently it is targeted as a hadoop 2.0.5 feature.
> (I'm a little surprised -- expected this to be a hadoop2 compat breaking
>feature) -- so I agree that this is a bit more urgent.
>
>Anyway, I agree that the fs .snapshot naming convention is long standing
>and should win.
>
>My concern is with breaking compatibility in 0.94 again -- if we don't go
>down the conf variable route,  I consider having docs to properly document
>how to do the upgrade and caveats of doing the upgrade in the docs/release
>notes blocker to hbase 0.94.7.  (specifically mentioning from 0.94.6 to
>0.94.7, and to possibly to 0.95).
>
>Jon.
>
>On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> bq. Alternatively, we can detect the underlying Hadoop version, and use
>
>> either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>>
>> I think this would introduce more confusion, especially for operations.
>>
>> Cheers
>>
>> On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <[EMAIL PROTECTED]> wrote:
>>
>> > Because HDFS exposes the snapshots so that the normal file system
>> > operations are mapped inside snapshot dirs, I think HDFS reserving the
>> > .snapshot name makes sense. OTOH, nothing is specific about the dir name
>> > that is chosen by HBase.
>> >
>> > I would prefer to change the dir name in 0.94 as well, since 0.94 is also
>> > being run on top of hadoop 2. Alternatively, we can detect the underlying
>> > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
>> > depending on h1 & h2.
>> >
>> > Enis
>> >
>> >
>> > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> >
>> > > bq. let's make the hbase snapshot for a conf variable.
>> > >
>> > > Once we decide on the new name of snapshot directory, we should still
>> use
>> > > hardcoded value. This aligns with current code base:
>> > > See this snippet from HConstants:
>> > >
>> > >   public static final List<String> HBASE_NON_TABLE_DIRS >> > >
>> > >     Collections.unmodifiableList(Arrays.asList(new String[] {
>> > > HREGION_LOGDIR_NAME,
>> > >
>> > >       HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>> > >
>> > >       HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
>> SNAPSHOT_DIR_NAME,
>> > > HBASE_TEMP_DIRECTORY }));
>> > > Cheers
>> > >
>> > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <[EMAIL PROTECTED]>
>> > wrote:
>> > >
>> > > > constraints:
>> > > >
>> > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
>> > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
>> will
>> > > ever
>> > > > make it to 1.x.  This hdfs feature ideally this shouldn't affect
>> > current
>> > > A
>> > > > pache Hbase 0.94.x's.
>> > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
>> > should
>> > > > pick a different table snapshot name to respect fs conventions.
>> > > >
>> > > > proposed actions:
>> > > >
>> > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
>> > > > snapshots.dir)  let's change the default for hbase 95+. (maybe
>> > > > .hbase-snapshots). we'll also port this patch to 0.94.x
>> > > > 2) let's publish instructions on how to update the hbase snapshot
>> dir:
>> > > > shutdown hbase, config update, rename dir, restart hbase.