Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Rewriting in image resizing program in terms of map reduce


Copy link to this message
-
Re: Rewriting in image resizing program in terms of map reduce
Hi Daniel,

I think there are better solutions, but simply chop the input file into
pieces ( i.e. 10 urls per file ) shall work.

2009/12/4 Daniel Garcia <[EMAIL PROTECTED]>

> Hello!
>
> I'm trying to rewrite an image resizing program in terms of map/reduce. The
> problem I see is that the job is not broken up in to small enough tasks. If
> I only have 1 input file with 10,000 urls (the file is much less than the
> HDFS block size) how can I ensure that the job is distributed amongst all
> the nodes. In other words how can I ensure that the task size is  small
> enough so that all nodes process a proportional size of the input.
>
> Regards,
> Daniel
>
>
Best Regards,

Tex Xu
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB