Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> File Integrity in HDFS


+
Stuti Awasthi 2012-05-01, 09:52
Copy link to this message
-
Re: File Integrity in HDFS
The easiest way out would be to rename files to pick-up-able name upon
successful copy, or have the loading done to a different directory and
rename/move the file when successfully closed to the job input
directory.

On Tue, May 1, 2012 at 3:22 PM, Stuti Awasthi <[EMAIL PROTECTED]> wrote:
> Hi All,
>
>
>
> I have a scenario in which Input files are copied to HDFS and MR jobs run on
> the input directory.
>
> Now there can be a scenario in which file is getting copied to HDFS and MR
> jobs starts , in this case I do not want my MR job to pick those files which
> are getting copied to hdfs and process of copying is not complete.
>
>
>
> Is there any way/api to check that if the file is not completely written to
> HDFS we can know.
>
>
>
> Regards,
>
> Stuti Awasthi
>
> HCL Comnet Systems and Services Ltd
>
> F-8/9 Basement, Sec-3,Noida.
>
>
>
>
> ________________________________
>
>
> ::DISCLAIMER::
> -----------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission cannot be guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or contain viruses.The e mail
> and its contents (with or without referred
> errors) shall therefore not attach any liability on the originator or HCL or
> its affiliates. Any views or opinions
> presented in this email are solely those of the author and may not
> necessarily reflect the opinions of HCL or its
> affiliates. Any form of reproduction, dissemination, copying, disclosure,
> Modification, distribution and/or publication
> of this message without the prior written consent of the author of this
> e-mail is strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately. Before opening any mail and attachments
> please check them for viruses and defect.
>
> -----------------------------------------------------------------------------------------------------------------------

--
Harsh J
+
Stuti Awasthi 2012-05-01, 12:23
+
Harsh J 2012-05-01, 12:41
+
Stuti Awasthi 2012-05-02, 15:50
+
Harsh J 2012-05-02, 18:32
+
Stuti Awasthi 2012-05-03, 05:05
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB