| clear query|facets|time |
Search criteria: .
Results from 41 to 50 from
142 (0.246s).
|
|
|
Loading phrases to help you
refine your search...
|
|
Re: Slow shuffle stage? - Hadoop - [mail # user]
|
|
...892 nodes, 4 tasks each, 3:1 mapper/reducer ratio. Each map task outputs four records, ~18MB each. They are fairly evenly distributed to the 17 reducers. As to ...
|
|
|
Author: Keith Wiley,
2011-11-11, 15:52
|
|
|
Slow shuffle stage? - Hadoop - [mail # user]
|
|
...What sorts of causes might be responsible for a long or slow shuffle stage? For example, I have a job of 266 maps (each emitting 4 records) and 17 reduces (each ingesting a...
|
|
|
Author: Keith Wiley,
2011-11-11, 01:20
|
|
|
Streaming input, data locality - Hadoop - [mail # user]
|
|
...I would like my streaming job to receive the names of files stored on HDFS, but not the actual contents of the files, and I would like data locality to be honored (I want mappers...
|
|
|
Author: Keith Wiley,
2011-08-19, 21:46
|
|
|
cmdenv LD_LIBRARY_PATH - Hadoop - [mail # user]
|
|
...I know you can do something like this: -cmdenv LD_LIBRARY_PATH=./my_libs if you have shared libraries in a subdirectory under the cwd (such as occurs when using -cacheArchi...
|
|
|
Author: Keith Wiley,
2011-08-05, 18:37
|
|
|
Re: streaming cacheArchive shared libraries - Hadoop - [mail # user]
|
|
...Right, so it was pushed down a level into the "testlink" directory. That's why my shared libraries were not linking properly to my mapper executable. I can fix that by usi...
|
|
|
Author: Keith Wiley,
2011-08-05, 17:49
|
|
|
Re: streaming cacheArchive shared libraries - Hadoop - [mail # user]
|
|
...Okay, I think I understand. The symlink name that follows the pound sign in the -cacheArchive directive isn't the name of the transferred jar file -- it is the name of a di...
|
|
|
Author: Keith Wiley,
2011-08-05, 17:48
|
|
|
Re: streaming cacheArchive shared libraries - Hadoop - [mail # user]
|
|
...Quick followup. I substituted the true mapper for a little python script that just lists the cwd's contents and dumps them to the streaming output (stderr). Oddly, I ...
|
|
|
Author: Keith Wiley,
2011-08-05, 17:27
|
|
|
streaming cacheArchive shared libraries - Hadoop - [mail # user]
|
|
...I can use cacheFile to load .so files into the distributed cache and it works fine (the streaming executable links against the .so and runs), but I can't get it to work with -cac...
|
|
|
Author: Keith Wiley,
2011-08-05, 17:10
|
|
|
Re: Upload, then decompress archive on HDFS? - Hadoop - [mail # user]
|
|
...I can envision an M/R job for the purpose of manipulating hdfs, such as (de)compressing files and resaving them back to HDFS. I just didn't think it should be necessary to ...
|
|
|
Author: Keith Wiley,
2011-08-05, 15:14
|
|
|
Upload, then decompress archive on HDFS? - Hadoop - [mail # user]
|
|
...Instead of "hd fs -put" hundreds of files of X megs, I want to do it once on a gzipped (or zipped) archive, one file, much smaller total megs. Then I want to decompress the...
|
|
|
Author: Keith Wiley,
2011-08-04, 23:28
|
|
|
|
