Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - LocalDirAllocator and getLocalPathForWrite


Copy link to this message
-
Re: LocalDirAllocator and getLocalPathForWrite
Marc Sturlese 2011-01-05, 23:26

Hey Todd,

>>LocalDirAllocator is an internal-facing API and you shouldn't be using it
>>from user code. If you write into mapred.local.dir like this, you will end
>>up with conflicts between different tasks running from the same node

 I know it's a bit odd usage but the thing is that I need to create files in
the local file system, work in there with them amb after that upload them to
hdfs (I use the outputcomitter.) To avoid the conflicts you talk about, I
create a folder which looks like "mapred.local.dir"/taskId/attemptId and I
work there and aparently I am having no problems.

>>and there isn't usually a good reason to write to multiple drives from
>>within a task

When I said I had a cluster of one node, was just to try to clarify my doubt
and explain the example. My cluster is bigger than that actually and each
node has more than 1 phisical disk. To have multuple task running at the
same time is what I do. I would like each task to write just to a single
local disk but don't know how to do it.

>>The working directory of your MR task is already within one of the drives,

Is there a way to get a working directory in the local disk from the
reducer?
Could I do something similar to:
FileSystem fs = FileSystem.get(conf);
LocalFileSystem localFs = fs.getLocal(conf);
Path path = localFs.getWorkingDirectory();
I would apreciate if you can tell me a bit more about this.
I need to deal with these files just in local and want them copied to hdfs
just when I finish working with them.

Thanks in advance.
--
View this message in context: http://lucene.472066.n3.nabble.com/LocalDirAllocator-and-getLocalPathForWrite-tp2199517p2202221.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.