Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - snapshotexport to outside of cluster


Copy link to this message
-
Re: snapshotexport to outside of cluster
Ted Yu 2013-12-01, 15:04
What if there is a bad sector on the disk of remote server ?
Using hdfs, you're shielded from such scenario.

On Dec 1, 2013, at 6:56 AM, oc tsdb <[EMAIL PROTECTED]> wrote:

> We want to use that exported data in case of any disaster happens.
>
> Thanks
> -OC
>
>
> On Sun, Dec 1, 2013 at 2:25 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> Since the remote server doesn't have hdfs, what would the backup be used
>> for ?
>>
>> Cheers
>>
>> On Nov 30, 2013, at 10:34 PM, oc tsdb <[EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>>>> Can you clarify what 'without copying to local file system' means ?
>>>
>>> We want to take HBase tables backup.So we periodically take snapshots and
>>> export data to backup server which is not in cluster.
>>>
>>> Currently we are achieving this as mentioned below.
>>>
>>> 1.Create snapshots from HBase command line
>>>  lets say we have created snapshot for a table called 'hbase_tabel' and
>>> its snapshot name is 'hbase_tabel_snapshot'
>>> 2. Then export snapshot data to remote backup server which is not in
>>> cluster.We are achieving this in two steps.
>>>  a. Export snapshot to local file system(sample path /tmp/hbase_backup)
>>> from hdfs file system as mentioned below
>>>      hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
>>> hbase_tabel_snapshot -copy-to file:///tmp/hbase_backup -
>>>      mappers      16
>>>     Here we should have enough disc space in local file system to export
>>> snapshot from hdfs.
>>>   b.Transfer(scp) exported snapshot from local file system
>>> (/tmp/hbase_backup)
>>> to backup server which is not hadoop/HBase running
>>>      server.
>>>
>>> That means first we have to export snapshot to local system from hdfs
>> and
>>> then transfer ( scp ) to remote server.Instead of doing in two steps can
>> we
>>> directly export snapshot to remote server(which is not in cluster) like
>> how
>>> it is exporting to another cluster.
>>> .
>>> Thanks
>>> OC
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Nov 30, 2013 at 11:11 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>
>>>> HBASE-4618 is not related to snapshots.
>>>> Lars did some work in HBASE-5509, a sub-task of HBASE-4618.
>>>>
>>>> FYI
>>>>
>>>>
>>>> On Fri, Nov 29, 2013 at 9:31 PM, Yu Li <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> I guess this might be a use case of
>>>>> HBASE-7912<https://issues.apache.org/jira/browse/HBASE-7912>or
>>>>> HBASE-4618 <https://issues.apache.org/jira/browse/HBASE-4618>? Maybe
>>>> some
>>>>> backup/restore tools taking use of snapshot would be nice to have?
>>>>>
>>>>>
>>>>> On 30 November 2013 12:58, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> The remote server is in different cluster, right ?
>>>>>>
>>>>>> Looking at ExportSnapshot#ExportMapper#copyFile(), it directly copies
>>>>> data
>>>>>> to outputPath which is in the other cluster.
>>>>>>
>>>>>> Can you clarify what 'without copying to local file system' means ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Fri, Nov 29, 2013 at 8:49 PM, oc tsdb <[EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> thanks for your replay.
>>>>>>>
>>>>>>> We are able to export snapshot to another cluster But this is not
>>>> what
>>>>> we
>>>>>>> are looking for.
>>>>>>> We want to export snapshot to remote server without copying to local
>>>>> file
>>>>>>> system.
>>>>>>>
>>>>>>> Thanks
>>>>>>> OC
>>>>>>>
>>>>>>> On Fri, Nov 29, 2013 at 8:52 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>>>>>
>>>>>>>> Here is a sample command exporting to another cluster:
>>>>>>>>
>>>>>>>> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
>>>>>>>> gs_1383068263 -copy-to hdfs://experiment-1d:8020/hbase -mappers 6
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Nov 29, 2013 at 4:02 AM, oc tsdb <[EMAIL PROTECTED]>
>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> can someone please let us know if we can export snapshots data
>>>>>> directly
>>>>>>>> to
>>>>>>>>> outside of cluster without copying to local.
>>>>>>>>>
>>>>>>>>> Please refer us if there are any blog with example.