Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> MapReduce on Local files


+
Mohammad Tariq 2013-04-02, 11:46
+
Lance Norskog 2013-04-05, 03:59
+
Harsh J 2013-04-03, 02:15
Copy link to this message
-
Re: MapReduce on Local files
Hello Harsh,

        Thank you for the response. I am sorry for being unclear.
Actually I was talking about the backup files which end with "~"
I mean these files are not visible normally, but my job is able to
see them. Does FileInputFormat behave in the same way for "~"
as it does in the case of "." and "_"?

Thanks.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Apr 3, 2013 at 7:45 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Not quite sure if I got your question. These tidbits may help though,
> from what I can understand:
>
> * LocalFileSystem's listing uses Java's APIs for file/dir listing, and
> has no concept of what a hidden file is on its own. It retrieves the
> whole list.
> * MR's FileInputFormat (and normal derivatives) does filter away "."
> and "_" starting path names, from added input paths to the job.
>
> On Wed, Apr 3, 2013 at 3:09 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > ---------- Forwarded message ----------
> > From: Mohammad Tariq <[EMAIL PROTECTED]>
> > Date: Tue, Apr 2, 2013 at 5:16 PM
> > Subject: MapReduce on Local files
> > To: [EMAIL PROTECTED]
> >
> >
> > Hello list,
> >
> >            Is a MR job capable of reading even the hidden temp files
> present
> > inside a directory located on my local FS?I have noticed this thing today
> > for the first time because till now I never tried running MR jobs on
> local
> > files.
> >
> > Thank you so much for your time?
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
>
>
>
> --
> Harsh J
>
+
Harsh J 2013-04-03, 09:58
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB