Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: best way to join?


+
Mirko Kämpf 2012-09-09, 09:55
+
Ted Dunning 2012-08-28, 20:07
+
dexter morgan 2012-08-30, 09:21
+
Ted Dunning 2012-08-30, 20:05
+
dexter morgan 2012-08-31, 13:03
+
dexter morgan 2012-09-02, 16:26
+
Ted Dunning 2012-09-03, 19:47
+
dexter morgan 2012-08-27, 20:15
+
Björn-Elmar Macek 2012-09-04, 08:17
Copy link to this message
-
Re: best way to join?
Ted Dunning 2012-08-27, 21:52
Mahout is getting some very fast knn code in version 0.8.

The basic work flow is that you would first do a large-scale clustering of
the data.  Then you would make a second pass using the clustering to
facilitate fast search for nearby points.

The clustering will require two map-reduce jobs, one to find the cluster
definitions and the second map-only to assign points to clusters in a form
to be used by the second pass.  The second pass is a map-only process as
well.

In order to make this as efficient as possible, it is desirable to use a
distribution of Hadoop that allows you to directly map the cluster data
structures into shared memory.  IF you have NFS access to your cluster,
this is easy.  Otherwise, it is considerably trickier.

On Mon, Aug 27, 2012 at 4:15 PM, dexter morgan <[EMAIL PROTECTED]>wrote:

> Dear list,
>
> Lets say i have a file, like this:
> id \t at,tlng <-- structure
>
> 1\t40.123,-50.432
> 2\t41.431,-43.32
> ...
> ...
> lets call it: 'points.txt'
> I'm trying to build a map-reduce job that runs over this BIG points file
> and it should output
> a file, that will look like:
> id[lat,lng] \t [list of points in JSON standard] <--- structure
>
> 1[40.123,-50.432]\t[[41.431,-43.32],[...,...],...,[...]]
> 2[41.431,-43.32]\t[[40.123,-50.432],...[,]]
> ...
>
> Basically it should run on ITSELF, and grab for each point the N (it will
> be an argument for the job) CLOSEST points (the mappers should calculate
> the distance)..
>
> Distributed cache is not an option, what else?  not sure if to classify it
> as a map-join , reduce-join or both?
> Would you do this in HIVE some how?
> Is it feasible in a single job?
>
> Would LOVE to hear your suggestions, code (if you think its not that hard)
> or what not.
> BTW using CDH3 - rev 3 (20.23)
>
> Thanks!
>
+
dexter morgan 2012-08-28, 13:48