Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> MapReduce on Local files


+
Mohammad Tariq 2013-04-02, 11:46
+
Lance Norskog 2013-04-05, 03:59
Copy link to this message
-
Re: MapReduce on Local files
Not quite sure if I got your question. These tidbits may help though,
from what I can understand:

* LocalFileSystem's listing uses Java's APIs for file/dir listing, and
has no concept of what a hidden file is on its own. It retrieves the
whole list.
* MR's FileInputFormat (and normal derivatives) does filter away "."
and "_" starting path names, from added input paths to the job.

On Wed, Apr 3, 2013 at 3:09 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> ---------- Forwarded message ----------
> From: Mohammad Tariq <[EMAIL PROTECTED]>
> Date: Tue, Apr 2, 2013 at 5:16 PM
> Subject: MapReduce on Local files
> To: [EMAIL PROTECTED]
>
>
> Hello list,
>
>            Is a MR job capable of reading even the hidden temp files present
> inside a directory located on my local FS?I have noticed this thing today
> for the first time because till now I never tried running MR jobs on local
> files.
>
> Thank you so much for your time?
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>

--
Harsh J
+
Mohammad Tariq 2013-04-03, 09:46
+
Harsh J 2013-04-03, 09:58