Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: MapReduce on Local files


Copy link to this message
-
Re: MapReduce on Local files
I see.
Thank you so much the clarification Harsh :)

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Apr 3, 2013 at 3:28 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> You've been misled by the GUI you use, I'm afraid. Many DEs (Desktop
> Environments) consider ~-suffix files as hidden but not the general
> standards (try ls for example, or even shell expansions, it will
> ignore . prefixes, but not ~ suffixes) :)
>
> To answer specifically though, no, the base FileInputFormat does not
> recognize ~ today, but if you want it to, you can pass a custom path
> filter to your InputFormat's implementation for when it calls the
> listStatus method.
>
> On Wed, Apr 3, 2013 at 3:16 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> > Hello Harsh,
> >
> >         Thank you for the response. I am sorry for being unclear.
> > Actually I was talking about the backup files which end with "~"
> > I mean these files are not visible normally, but my job is able to
> > see them. Does FileInputFormat behave in the same way for "~"
> > as it does in the case of "." and "_"?
> >
> > Thanks.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Wed, Apr 3, 2013 at 7:45 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>
> >> Not quite sure if I got your question. These tidbits may help though,
> >> from what I can understand:
> >>
> >> * LocalFileSystem's listing uses Java's APIs for file/dir listing, and
> >> has no concept of what a hidden file is on its own. It retrieves the
> >> whole list.
> >> * MR's FileInputFormat (and normal derivatives) does filter away "."
> >> and "_" starting path names, from added input paths to the job.
> >>
> >> On Wed, Apr 3, 2013 at 3:09 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > Warm Regards,
> >> > Tariq
> >> > https://mtariq.jux.com/
> >> > cloudfront.blogspot.com
> >> >
> >> >
> >> > ---------- Forwarded message ----------
> >> > From: Mohammad Tariq <[EMAIL PROTECTED]>
> >> > Date: Tue, Apr 2, 2013 at 5:16 PM
> >> > Subject: MapReduce on Local files
> >> > To: [EMAIL PROTECTED]
> >> >
> >> >
> >> > Hello list,
> >> >
> >> >            Is a MR job capable of reading even the hidden temp files
> >> > present
> >> > inside a directory located on my local FS?I have noticed this thing
> >> > today
> >> > for the first time because till now I never tried running MR jobs on
> >> > local
> >> > files.
> >> >
> >> > Thank you so much for your time?
> >> >
> >> > Warm Regards,
> >> > Tariq
> >> > https://mtariq.jux.com/
> >> > cloudfront.blogspot.com
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB