Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Memory mapped resources

Copy link to this message
Re: Memory mapped resources
Ted Dunning 2011-04-12, 15:35
Blocks live where they land when first created.  They can be moved due to
node failure or rebalancing, but it is typically pretty expensive to do
this.  It certainly is slower than just reading the file.

If you really, really want mmap to work, then you need to set up some native
code that builds an mmap'ed region, but sets all pages to no access if the
corresponding block is non-local and sets the block to access the local
block if the block is local. Then you can intercept the segmentation
violations that occur on page access to non-local data, read that block to
local storage and mmap it into those pages.

This is LOTS of work to get exactly right and must be done in C since Java
can't really handle seg faults correctly.  This pattern is fairly commonly
used in garbage collected languages to allow magical remapping of memory
without explicit tests.

On Tue, Apr 12, 2011 at 8:24 AM, Jason Rutherglen <

> Interesting.  I'm not familiar with how blocks go local, however I'm
> interested in how to make this occur via a manual oriented call.  Eg,
> is there an option available that guarantees locality, and if not,
> perhaps there's work being done towards that path?