Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - How to tune fileSystem.listFiles("/", true) if you like walk though almost all files


Copy link to this message
-
How to tune fileSystem.listFiles("/", true) if you like walk though almost all files
Christian Schneider 2013-08-12, 19:33
Hi, is there a way to tune this?

I walk though the files with:

RemoteIterator<LocatedFileStatus> listFiles = fileSystem.listFiles(new
Path(uri), true);
while(listFiles.hasNext()) {
    listFiles.next();
};

I need to get some information about those files, therefore i like to scan
them all.

Is there any way to tune the listFiles.next() call. Like loading a bunch of
files, or multithreading it?

Best Regards,
Christian.