Hello All,

Is there any clean way to tell from the API (v0.20.2) that a file in HDFS is currently being written to?  I've seen some exceptions before related to it, but I was hoping there is a clean way and Google isn't turning anything up for me.

Thanks!
-- Adam

On 08/17/2011 12:57 PM, Adam Shook wrote:
> Hello All,
>
> Is there any clean way to tell from the API (v0.20.2) that a file in HDFS is currently being written to?  I've seen some exceptions before related to it, but I was hoping there is a clean way and Google isn't turning anything up for me.
>
> Thanks!
> -- Adam

You might be able to do it to some extent using
FileStatus.getModificationTime()
(http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileStatus.html#getModificationTime()),
but this would really be a hack, IMO, and not something you should rely on.

I think you'd be better off either a) writing the file to a temp
directory, or b) writing it with a .tmp extension, and then moving or
renaming it once the file write is complete.

HTH,

DR

Sadly, I don't have control over naming the files.  They are being ingested in HDFS by powers out of my control.  I'll mess around with the modification times and see if I can get a good solution.  If anyone knows of a way that seems less hackish, I am all ears.

Thanks, Adam
-----Original Message-----
From: David Rosenstrauch [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, August 17, 2011 1:22 PM
To: [EMAIL PROTECTED]
Subject: Re: HDFS File being written

On 08/17/2011 12:57 PM, Adam Shook wrote:
> Hello All,
>
> Is there any clean way to tell from the API (v0.20.2) that a file in HDFS is currently being written to?  I've seen some exceptions before related to it, but I was hoping there is a clean way and Google isn't turning anything up for me.
>
> Thanks!
> -- Adam

You might be able to do it to some extent using
FileStatus.getModificationTime()
(http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileStatus.html#getModificationTime()),
but this would really be a hack, IMO, and not something you should rely on.

I think you'd be better off either a) writing the file to a temp
directory, or b) writing it with a .tmp extension, and then moving or
renaming it once the file write is complete.

HTH,

DR

-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1392 / Virus Database: 1520/3840 - Release Date: 08/17/11

Just my $0.02, but I think you really ought to push back to have
whoever's creating the files upstream do it in one of the manners I
described.  This way will be way too error prone.  I mean, think about
it:  with your current set up you not only can't reliably know if the
creator is finished writing to a file, but you also can't know if even
if they are finished whether the file write was completed successfully.
  The creator could have aborted the file write in the middle - either
purposely or inadvertently - and you'll be trying to process an
incomplete file.

You really need to employ *some* method to reliably determine when a
file is successfully uploaded, or you're going to wind up with a very
buggy system.

DR

On 08/17/2011 01:41 PM, Adam Shook wrote:
> Sadly, I don't have control over naming the files.  They are being ingested in HDFS by powers out of my control.  I'll mess around with the modification times and see if I can get a good solution.  If anyone knows of a way that seems less hackish, I am all ears.
>
> Thanks, Adam
>
>
>
>
> -----Original Message-----
> From: David Rosenstrauch [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, August 17, 2011 1:22 PM
> To: [EMAIL PROTECTED]
> Subject: Re: HDFS File being written
>
> On 08/17/2011 12:57 PM, Adam Shook wrote:
>> Hello All,
>>
>> Is there any clean way to tell from the API (v0.20.2) that a file in HDFS is currently being written to?  I've seen some exceptions before related to it, but I was hoping there is a clean way and Google isn't turning anything up for me.
>>
>> Thanks!
>> -- Adam
>
> You might be able to do it to some extent using
> FileStatus.getModificationTime()
> (http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileStatus.html#getModificationTime()),
> but this would really be a hack, IMO, and not something you should rely on.
>
> I think you'd be better off either a) writing the file to a temp
> directory, or b) writing it with a .tmp extension, and then moving or
> renaming it once the file write is complete.
>
> HTH,
>
> DR
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 10.0.1392 / Virus Database: 1520/3840 - Release Date: 08/17/11

Amen.

Without a solid hand-off, your system is going to be subject all kinds of
failure modes.

On Wed, Aug 17, 2011 at 11:17 AM, David Rosenstrauch <[EMAIL PROTECTED]>wrote:

> You really need to employ *some* method to reliably determine when a file
> is successfully uploaded, or you're going to wind up with a very buggy
> system.
>

Yes. Renaming the file or moving it will help you avoid a lot of issues. If
it's possible it will be even better to write a set of files to a temp
directory and then moving the directory to some staging/processing dir as it
would reduce the load on the NameNode.

Thanks
Sudhan S

On Wed, Aug 17, 2011 at 11:11 PM, Adam Shook <[EMAIL PROTECTED]> wrote:

> Sadly, I don't have control over naming the files.  They are being ingested
> in HDFS by powers out of my control.  I'll mess around with the modification
> times and see if I can get a good solution.  If anyone knows of a way that
> seems less hackish, I am all ears.
>
> Thanks, Adam
>
>
>
>
> -----Original Message-----
> From: David Rosenstrauch [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, August 17, 2011 1:22 PM
> To: [EMAIL PROTECTED]
> Subject: Re: HDFS File being written
>
> On 08/17/2011 12:57 PM, Adam Shook wrote:
> > Hello All,
> >
> > Is there any clean way to tell from the API (v0.20.2) that a file in HDFS
> is currently being written to?  I've seen some exceptions before related to
> it, but I was hoping there is a clean way and Google isn't turning anything
> up for me.
> >
> > Thanks!
> > -- Adam
>
> You might be able to do it to some extent using
> FileStatus.getModificationTime()
> (
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileStatus.html#getModificationTime()
> ),
> but this would really be a hack, IMO, and not something you should rely on.
>
> I think you'd be better off either a) writing the file to a temp
> directory, or b) writing it with a .tmp extension, and then moving or
> renaming it once the file write is complete.
>
> HTH,
>
> DR
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 10.0.1392 / Virus Database: 1520/3840 - Release Date: 08/17/11
>