Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> How to tune fileSystem.listFiles("/", true) if you like walk though almost all files


Copy link to this message
-
How to tune fileSystem.listFiles("/", true) if you like walk though almost all files
Hi, is there a way to tune this?

I walk though the files with:

RemoteIterator<LocatedFileStatus> listFiles = fileSystem.listFiles(new
Path(uri), true);
while(listFiles.hasNext()) {
    listFiles.next();
};

I need to get some information about those files, therefore i like to scan
them all.

Is there any way to tune the listFiles.next() call. Like loading a bunch of
files, or multithreading it?

Best Regards,
Christian.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB