Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> breadth-first search


+
Peng, Wei 2010-12-21, 01:49
+
Edward J. Yoon 2010-12-21, 02:22
+
Peng, Wei 2010-12-21, 02:30
+
Edward J. Yoon 2010-12-21, 06:27
+
Peng, Wei 2010-12-21, 09:59
+
Ted Dunning 2010-12-21, 15:28
+
Ricky Ho 2010-12-21, 03:00
+
Peng, Wei 2010-12-21, 04:16
+
Ted Dunning 2010-12-21, 05:02
+
Peng, Wei 2010-12-21, 05:43
+
Ted Dunning 2010-12-21, 06:18
+
Edward J. Yoon 2010-12-22, 06:54
+
Ricky Ho 2010-12-21, 16:53
+
Peng, Wei 2010-12-21, 18:04
+
Ted Dunning 2010-12-21, 18:09
+
Peng, Wei 2010-12-22, 02:07
+
Peng, Wei 2010-12-22, 16:57
+
Ricky Ho 2010-12-22, 18:46
+
Ted Dunning 2010-12-22, 19:00
+
Peng, Wei 2010-12-22, 19:51
Copy link to this message
-
Re: breadth-first search
Modify the following parameters:
mapred.tasktracker.map.tasks.maximum
mapred.tasktracker.reduce.tasks.maximum
mapred.map.tasks
mapred.reduce.tasks

FYI you need to adjust the -Xmx for your mapper/reducer after increasing the
values for above parameters

On Wed, Dec 22, 2010 at 11:51 AM, Peng, Wei <[EMAIL PROTECTED]> wrote:

> Thanks for quick response.
>
> Partitioning graphs into subgraphs and later combining the results is
> too complicated to do. I prefer a simple method.
>
> Currently, I do not want to divide the breadth-first search from a
> single source. I just want to run 100 breadth-first search from 100
> source nodes with 100 threads running in parallel.
> The problem is that these 100 threads do not seem to run parallel,
> however, they seem to run in sequential. I have searched on-line. Some
> people mention that all tasks are put into queues waiting for free
> mapreduce slots. It is might be due to not enough slots.
> How to deal with this problem?
>
> Wei
>
>
> -----Original Message-----
> From: Ted Dunning [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, December 22, 2010 2:01 PM
> To: [EMAIL PROTECTED]
> Subject: Re: breadth-first search
>
> The Mahout math package has a number of basic algorithms that use
> algorithmic efficiencies when given sparse graphs.
>
> A number of other algorithms use only the product of a sparse matrix on
> another matrix or a vector.  Since these algorithms never change the
> original sparse matrix, they are safe against fill-in problems.
>
> The random projection technique avoids O(v^3) algorithms for computing
> SVD
> or related matrix decompositions.  See http://arxiv.org/abs/0909.4061
> and
> https://issues.apache.org/jira/browse/MAHOUT-376
>
> None of these these algorithms are specific to graph theory, but all
> deal
> with methods that are useful with sparse graphs.
>
> On Wed, Dec 22, 2010 at 10:46 AM, Ricky Ho <[EMAIL PROTECTED]>
> wrote:
>
> > Can you point me to Matrix algorithms that is tuned for sparse graph ?
> >  What I
> > mean is from O(v^3) to O(v*e)  where v = number of vertex and e > number of
> > edges.
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB