| clear query|facets|time |
Search criteria: .
Results from 1 to 10 from
15 (0.117s).
|
|
|
Loading phrases to help you
refine your search...
|
|
Re: how to differentiate which input directory current record comes from? - MapReduce - [mail # user]
|
|
...If you use the new API, you can access the MapContext object in the setup method of the mapper. Then, you can get the input split with MapContext#getInputSplit(), cast it to FileSplit and ob...
|
|
|
Author: Sigurd Spieckermann,
2012-12-15, 10:17
|
|
|
Re: Spill file compression - MapReduce - [mail # user]
|
|
...When I log the calls of the combiner function and print the number of elements iterated over, it is all 1 during the spill-writing phase and the combiner is called very often. Is this normal...
|
|
|
Author: Sigurd Spieckermann,
2012-11-07, 15:14
|
|
|
Re: Spill file compression - MapReduce - [mail # user]
|
|
...Hm, maybe I need some clarification on what the combiner exactly does. From what I understand from "Hadoop - The Definitive Guide", there are a few occasions when a combiner may be called be...
|
|
|
Author: Sigurd Spieckermann,
2012-11-07, 14:29
|
|
|
Re: Spill file compression - MapReduce - [mail # user]
|
|
...OK, I found the answer to one of my questions just now -- the location of the spill files and their sizes. So, there's a discrepancy between what I see and what you said about the compressio...
|
|
|
Author: Sigurd Spieckermann,
2012-11-07, 13:18
|
|
|
Re: Spill file compression - MapReduce - [mail # user]
|
|
...OK, just wanted to confirm. Maybe there is another problem then. I just looked at the task logs and there were ~200 spills recorded for a single task, only afterwards there was a merge phase...
|
|
|
Author: Sigurd Spieckermann,
2012-11-07, 13:12
|
|
|
Spill file compression - MapReduce - [mail # user]
|
|
...Hi guys, I've encountered a situation where the ratio between "Map output bytes" and "Map output materialized bytes" is quite huge and during the map-phase data is spilled to disk quit...
|
|
|
Author: Sigurd Spieckermann,
2012-11-07, 12:32
|
|
|
Re: Data locality of map-side join - MapReduce - [mail # user]
|
|
...Interesting to know that Hive and Pig are doing something in this direction. I'm dealing with the Hadoop join-package which doesn't use DistributedCache though but it rather pulls the other ...
|
|
|
Author: Sigurd Spieckermann,
2012-10-23, 09:22
|
|
|
Data locality of map-side join - MapReduce - [mail # user]
|
|
...Hi guys, I've been trying to figure out whether a map-side join using the join-package does anything clever regarding data locality with respect to at least one of the part...
|
|
|
Author: Sigurd Spieckermann,
2012-10-22, 20:29
|
|
|
Join-package in new API? - MapReduce - [mail # user]
|
|
...Hi, I've just noticed that the join-package only exists in the old map-reduce API. Is there a particular reason why it's not in the new API? (deprecated maybe?) If so, what is the appr...
|
|
|
Author: Sigurd Spieckermann,
2012-10-10, 14:47
|
|
|
Re: Reading from HDFS from inside the mapper - MapReduce - [mail # user]
|
|
...OK, I see... Is there any way to change this? I need guaranteed order in order for the map-side join to work correctly and I need the standalone mode for debugging code that is executed on t...
|
|
|
Author: Sigurd Spieckermann,
2012-09-17, 13:50
|
|
|
|
