-Re: Data locality of map-side join
Bejoy KS 2012-10-23, 06:21
Mapside joins are efficiently implemented in Hive and Pig. I'm talking in terms of how mapside joins are implemented in hive.
In map side join, the smaller data set is first loaded into DistributedCache. The larger dataset is streamed as usual and the smaller dataset in memory. For every record in larger data set the look up is made in memory on the smaller set and there by joins are done.
In later versions of hive the hive framework itself intelligently determines the smaller data set. In older versions you can specify the smaller data set using some hints in query.
Sent from handheld, please excuse typos.
From: Sigurd Spieckermann <[EMAIL PROTECTED]>
Date: Mon, 22 Oct 2012 22:29:15
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Data locality of map-side join
I've been trying to figure out whether a map-side join using the
join-package does anything clever regarding data locality with respect
to at least one of the partitions to join. To be more specific, if I
want to join two datasets and some partition of dataset A is larger than
the corresponding partition of dataset B, does Hadoop account for this
and try to ensure that the map task is executed on the datanode storing
the bigger partition thus reducing data transfer (if the other partition
does not happen to be located on that same datanode)? I couldn't
conclude the one or the other behavior from the source code and I
couldn't find any documentation about this detail.
Thanks for clarifying!