Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - breadth-first search


+
Peng, Wei 2010-12-21, 01:49
+
Edward J. Yoon 2010-12-21, 02:22
+
Peng, Wei 2010-12-21, 02:30
+
Edward J. Yoon 2010-12-21, 06:27
+
Peng, Wei 2010-12-21, 09:59
+
Ted Dunning 2010-12-21, 15:28
+
Ricky Ho 2010-12-21, 03:00
+
Peng, Wei 2010-12-21, 04:16
+
Ted Dunning 2010-12-21, 05:02
+
Peng, Wei 2010-12-21, 05:43
+
Ted Dunning 2010-12-21, 06:18
+
Edward J. Yoon 2010-12-22, 06:54
+
Ricky Ho 2010-12-21, 16:53
+
Peng, Wei 2010-12-21, 18:04
+
Ted Dunning 2010-12-21, 18:09
+
Peng, Wei 2010-12-22, 02:07
+
Peng, Wei 2010-12-22, 16:57
+
Ricky Ho 2010-12-22, 18:46
+
Ted Dunning 2010-12-22, 19:00
+
Peng, Wei 2010-12-22, 19:51
Copy link to this message
-
Re: breadth-first search
Ted Yu 2010-12-22, 20:35
Modify the following parameters:
mapred.tasktracker.map.tasks.maximum
mapred.tasktracker.reduce.tasks.maximum
mapred.map.tasks
mapred.reduce.tasks

FYI you need to adjust the -Xmx for your mapper/reducer after increasing the
values for above parameters

On Wed, Dec 22, 2010 at 11:51 AM, Peng, Wei <[EMAIL PROTECTED]> wrote:

> Thanks for quick response.
>
> Partitioning graphs into subgraphs and later combining the results is
> too complicated to do. I prefer a simple method.
>
> Currently, I do not want to divide the breadth-first search from a
> single source. I just want to run 100 breadth-first search from 100
> source nodes with 100 threads running in parallel.
> The problem is that these 100 threads do not seem to run parallel,
> however, they seem to run in sequential. I have searched on-line. Some
> people mention that all tasks are put into queues waiting for free
> mapreduce slots. It is might be due to not enough slots.
> How to deal with this problem?
>
> Wei
>
>
> -----Original Message-----
> From: Ted Dunning [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, December 22, 2010 2:01 PM
> To: [EMAIL PROTECTED]
> Subject: Re: breadth-first search
>
> The Mahout math package has a number of basic algorithms that use
> algorithmic efficiencies when given sparse graphs.
>
> A number of other algorithms use only the product of a sparse matrix on
> another matrix or a vector.  Since these algorithms never change the
> original sparse matrix, they are safe against fill-in problems.
>
> The random projection technique avoids O(v^3) algorithms for computing
> SVD
> or related matrix decompositions.  See http://arxiv.org/abs/0909.4061
> and
> https://issues.apache.org/jira/browse/MAHOUT-376
>
> None of these these algorithms are specific to graph theory, but all
> deal
> with methods that are useful with sparse graphs.
>
> On Wed, Dec 22, 2010 at 10:46 AM, Ricky Ho <[EMAIL PROTECTED]>
> wrote:
>
> > Can you point me to Matrix algorithms that is tuned for sparse graph ?
> >  What I
> > mean is from O(v^3) to O(v*e)  where v = number of vertex and e > number of
> > edges.
> >
>