Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - number of reducers


Copy link to this message
-
number of reducers
jamal sasha 2012-11-20, 19:38
Hi,

  I wrote a simple map reduce job in hadoop streaming.

I am wondering if I am doing something wrong ..

While number of mappers are projected to be around 1700.. reducers.. just 1?

It’s couple of TB’s worth of data.

What can I do to address this.

Basically mapper looks like this

For line in sys.stdin:

    Print line

Reducer

For line in sys.stdin:

    New_line = process_line(line)

    Print new_line

Thanks
+
Bejoy KS 2012-11-20, 20:09
+
Kartashov, Andy 2012-11-20, 21:50
+
alxsss@... 2012-11-20, 22:00
+
jamal sasha 2012-11-20, 20:24
+
Harsh J 2012-11-21, 04:08