Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> File Integrity in HDFS


+
Stuti Awasthi 2012-05-01, 09:52
Copy link to this message
-
Re: File Integrity in HDFS
The easiest way out would be to rename files to pick-up-able name upon
successful copy, or have the loading done to a different directory and
rename/move the file when successfully closed to the job input
directory.

On Tue, May 1, 2012 at 3:22 PM, Stuti Awasthi <[EMAIL PROTECTED]> wrote:
> Hi All,
>
>
>
> I have a scenario in which Input files are copied to HDFS and MR jobs run on
> the input directory.
>
> Now there can be a scenario in which file is getting copied to HDFS and MR
> jobs starts , in this case I do not want my MR job to pick those files which
> are getting copied to hdfs and process of copying is not complete.
>
>
>
> Is there any way/api to check that if the file is not completely written to
> HDFS we can know.
>
>
>
> Regards,
>
> Stuti Awasthi
>
> HCL Comnet Systems and Services Ltd
>
> F-8/9 Basement, Sec-3,Noida.
>
>
>
>
> ________________________________
>
>
> ::DISCLAIMER::
> -----------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission cannot be guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or contain viruses.The e mail
> and its contents (with or without referred
> errors) shall therefore not attach any liability on the originator or HCL or
> its affiliates. Any views or opinions
> presented in this email are solely those of the author and may not
> necessarily reflect the opinions of HCL or its
> affiliates. Any form of reproduction, dissemination, copying, disclosure,
> Modification, distribution and/or publication
> of this message without the prior written consent of the author of this
> e-mail is strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately. Before opening any mail and attachments
> please check them for viruses and defect.
>
> -----------------------------------------------------------------------------------------------------------------------

--
Harsh J
+
Stuti Awasthi 2012-05-01, 12:23
+
Harsh J 2012-05-01, 12:41
+
Stuti Awasthi 2012-05-02, 15:50
+
Harsh J 2012-05-02, 18:32
+
Stuti Awasthi 2012-05-03, 05:05