Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> set mapred.map.tasks=1 not work


Copy link to this message
-
Re: set mapred.map.tasks=1 not work
I have lots of small files in hive, the mapred is too slow .... Is there a
way to improve the speed ?

2010/6/10 Edward Capriolo <[EMAIL PROTECTED]>

>
>
> On Wed, Jun 9, 2010 at 3:04 AM, wd <[EMAIL PROTECTED]> wrote:
>
>> I've tried hive 0.5, the option not work too.
>> And find this page[
>> http://markmail.org/message/k32nrcb2ncsq67ef?q=mapred.map.tasks+#query:mapred.map.tasks%20+page:1+mid:k32nrcb2ncsq67ef+state:results]
>> via google.
>>
>> 2010/6/9 wd <[EMAIL PROTECTED]>
>>
>> hi,
>>>
>>> I'm using hive svn rev946854. And try to set mapred.map.tasks=1 at hive
>>> cli, but seemes it doesn't work, total map tasks still over 300+.
>>>
>>> Is this a svn version problem?
>>>
>>
>>
> You answered your own question, look in the link
>
> "You cannot force *mapred.map.tasks* but can specify mapred.reduce.tasks.
> "
>
> Map tasks is based on the number of input files and folders. Even though
> hive uses a CombinedInput format you still can get a number of mappers.
>
> Edward
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB