Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Review Request: FLUME-1699. Make the rename of the meta file platform neutral


+
Hari Shreedharan 2012-11-16, 19:26
+
Roshan Naik 2012-11-16, 19:48
+
Hari Shreedharan 2012-11-16, 20:08
+
Mike Percy 2012-11-16, 20:45
Copy link to this message
-
Re: Review Request: FLUME-1699. Make the rename of the meta file platform neutral


> On Nov. 16, 2012, 7:48 p.m., Roshan Naik wrote:
> > why not simplify it to ..
> >
> >
> > //  delete log.meta  
> > //  log.meta.tmp -> log.meta
> >
>
> Hari Shreedharan wrote:
>     I am not entirely sure how all platforms handle renames. Depending on implementation, though highly unlikely, if the system crashes in the middle of a rename, the tmp file might disappear without the ".meta" file being created. Leaving the old file guarantees that either the old or the new file is available at any point in time.
>    
>     I am just being paranoid about it since I don't know how different operating systems implement renames (for example, Windows does not seem to give many guarantees about durability in the event of a crash during a rename)
>
> Mike Percy wrote:
>     Do we ever look for log.meta.old? If not, I think we should look for it @ startup time if log.meta does not exist.
>    
>     Roshan: If we lose the meta file on an encrypted file channel, we are screwed because the meta file contains the initialization vector for AES/CTR/Nopadding. The data will be irrecovably lost.
>

Hmm, you are right. I was just think of leaving it as is, and expect that the user would fix it by doing the renames manually as is. Will update the patch. Thanks Mike, Roshan.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8087/#review13525
-----------------------------------------------------------
On Nov. 16, 2012, 7:26 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8087/
> -----------------------------------------------------------
>
> (Updated Nov. 16, 2012, 7:26 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> If atomic rename fails, rename the current meta file to old, rename tmp to meta and then delete old.
>
>
> This addresses bug FLUME-1699.
>     https://issues.apache.org/jira/browse/FLUME-1699
>
>
> Diffs
> -----
>
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java f768d23
>
> Diff: https://reviews.apache.org/r/8087/diff/
>
>
> Testing
> -------
>
> All unit tests pass
>
>
> Thanks,
>
> Hari Shreedharan
>
>

+
Hari Shreedharan 2012-11-19, 04:42
+
Hari Shreedharan 2012-11-19, 04:53
+
Brock Noland 2012-11-19, 18:54
+
Hari Shreedharan 2012-11-19, 18:56
+
Hari Shreedharan 2012-11-19, 19:41
+
Brock Noland 2012-11-19, 19:47
+
Hari Shreedharan 2012-11-19, 19:56
+
Brock Noland 2012-11-19, 20:26