Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - number of reducers


Copy link to this message
-
number of reducers
jamal sasha 2012-11-20, 19:38
Hi,

  I wrote a simple map reduce job in hadoop streaming.

I am wondering if I am doing something wrong ..

While number of mappers are projected to be around 1700.. reducers.. just 1?

It’s couple of TB’s worth of data.

What can I do to address this.

Basically mapper looks like this

For line in sys.stdin:

    Print line

Reducer

For line in sys.stdin:

    New_line = process_line(line)

    Print new_line

Thanks