Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - MapReduce on Local files


+
Mohammad Tariq 2013-04-02, 11:46
+
Lance Norskog 2013-04-05, 03:59
+
Harsh J 2013-04-03, 02:15
+
Mohammad Tariq 2013-04-03, 09:46
Copy link to this message
-
Re: MapReduce on Local files
Harsh J 2013-04-03, 09:58
You've been misled by the GUI you use, I'm afraid. Many DEs (Desktop
Environments) consider ~-suffix files as hidden but not the general
standards (try ls for example, or even shell expansions, it will
ignore . prefixes, but not ~ suffixes) :)

To answer specifically though, no, the base FileInputFormat does not
recognize ~ today, but if you want it to, you can pass a custom path
filter to your InputFormat's implementation for when it calls the
listStatus method.

On Wed, Apr 3, 2013 at 3:16 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello Harsh,
>
>         Thank you for the response. I am sorry for being unclear.
> Actually I was talking about the backup files which end with "~"
> I mean these files are not visible normally, but my job is able to
> see them. Does FileInputFormat behave in the same way for "~"
> as it does in the case of "." and "_"?
>
> Thanks.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, Apr 3, 2013 at 7:45 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> Not quite sure if I got your question. These tidbits may help though,
>> from what I can understand:
>>
>> * LocalFileSystem's listing uses Java's APIs for file/dir listing, and
>> has no concept of what a hidden file is on its own. It retrieves the
>> whole list.
>> * MR's FileInputFormat (and normal derivatives) does filter away "."
>> and "_" starting path names, from added input paths to the job.
>>
>> On Wed, Apr 3, 2013 at 3:09 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > ---------- Forwarded message ----------
>> > From: Mohammad Tariq <[EMAIL PROTECTED]>
>> > Date: Tue, Apr 2, 2013 at 5:16 PM
>> > Subject: MapReduce on Local files
>> > To: [EMAIL PROTECTED]
>> >
>> >
>> > Hello list,
>> >
>> >            Is a MR job capable of reading even the hidden temp files
>> > present
>> > inside a directory located on my local FS?I have noticed this thing
>> > today
>> > for the first time because till now I never tried running MR jobs on
>> > local
>> > files.
>> >
>> > Thank you so much for your time?
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>>
>>
>>
>> --
>> Harsh J
>
>

--
Harsh J