| clear query|facets|time |
Search criteria: .
Results from 1 to 9 from
9 (0.253s).
|
|
|
Loading phrases to help you
refine your search...
|
|
Re: Question on HDFS_BYTES_READ and HDFS_BYTES_WRITTEN - MapReduce - [mail # user]
|
|
...Hello Harsh, Thanks for the useful feedback. You were right. My map tasks open additional files from hdfs. The catch was that I had thousands of map tasks being created and each of the...
|
|
|
Author: Jim Twensky,
2013-05-17, 21:50
|
|
|
Question on HDFS_BYTES_READ and HDFS_BYTES_WRITTEN - MapReduce - [mail # user]
|
|
...I have an iterative MapReduce job that I run over 35 GB of data repeatedly. The output of the first job is the input to the second one and it goes on like that until convergence. I am ...
|
|
|
Author: Jim Twensky,
2013-05-14, 17:11
|
|
|
Re: Wrapping around BitSet with the Writable interface - MapReduce - [mail # user]
|
|
...Thanks for the suggestions. I ended up switching to jdk 1.7+ just to make the code more readable. I will take a look at the EWAH implementation as well. Jim On Sun, May 12, 2013...
|
|
|
Author: Jim Twensky,
2013-05-13, 15:51
|
|
|
Wrapping around BitSet with the Writable interface - MapReduce - [mail # user]
|
|
...I have large java.util.BitSet objects that I want to bitwise-OR using a MapReduce job. I decided to wrap around each object using the Writable interface. Right now I convert each BitSet to a...
|
|
|
Author: Jim Twensky,
2013-05-12, 18:24
|
|
|
Redundant synchronization in ReduceTask.java - MapReduce - [mail # dev]
|
|
...Hi, I have recently been reading Hadoop 1.1.0 source code to better understand the internals and learned a lot from it, so far. When I was looking at ReduceTask.java, I saw some synchr...
|
|
|
Author: Jim Twensky,
2012-12-08, 20:58
|
|
|
Re: Chaning Multiple Reducers: Reduce -> Reduce -> Reduce - MapReduce - [mail # user]
|
|
...Thank you for the comments. Some similar frameworks I looked at include Haloop, Twister, Hama, Giraph and Cascading. I am also doing large scale graph processing so I assumed one of them cou...
|
|
|
Author: Jim Twensky,
2012-10-08, 19:09
|
|
|
Re: Chaning Multiple Reducers: Reduce -> Reduce -> Reduce - MapReduce - [mail # user]
|
|
...Hi Harsh, The hidden map operation which is applied to the reduced partition at one stage can generate keys that are outside of the range covered by that particular reducer. I still ne...
|
|
|
Author: Jim Twensky,
2012-10-05, 18:02
|
|
|
Re: Chaning Multiple Reducers: Reduce -> Reduce -> Reduce - MapReduce - [mail # user]
|
|
...Hi Harsh, Yes, there is actually a "hidden" map stage, that generates new pairs based on the last reduce output but I can create those records during the reduce step instead and ...
|
|
|
Author: Jim Twensky,
2012-10-05, 17:43
|
|
|
Chaning Multiple Reducers: Reduce -> Reduce -> Reduce - MapReduce - [mail # user]
|
|
...Hi, I have a complex Hadoop job that iterates over large graph data multiple times until some convergence condition is met. I know that the map output goes to the local disk of e...
|
|
|
Author: Jim Twensky,
2012-10-05, 16:31
|
|
|
|
