Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Terasort


On Mon, May 14, 2012 at 10:40 AM, Barry, Sean F <[EMAIL PROTECTED]> wrote:
> I am having a bit of trouble understanding how the Terasort benchmark works, especially the fundamentals of how the data is sorted. If the data is being split into many chunks wouldn't it all have to be re-integrated back into the entire dataset?

Before the job is launched, the input is sampled to find "cut" points.
Those cut points are used to assign keys to reduces. For example, if
you have 100 reduces, there are 99 keys chosen. All keys less than the
first are sent to the first reduce, between the first two keys are
sent to the second reduce and so on. The logic is done by the
TotalOrderPartitioner, which replaces MapReduce's default
HashPartitioner.

-- Owen