Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Proper utilization of Map and reduce


Copy link to this message
-
Re: Proper utilization of Map and reduce
Hi,

Can this be because of the scheduler you are using? I see there are 3
queues, check the scheduler configuration, that might give you some hints.

Thanks,
Aniket

On Thu, Jan 12, 2012 at 12:24 AM, hadoop hive <[EMAIL PROTECTED]> wrote:

> thanks for your reply bejoy
>
> Actually there is nothing like rack awareness(actually they are on default
> rack), sometime there are also some reducers when required by query, i m
> running jobs through hive cli,
>
> you can suggest me some R&D so that i can check wat is actually going on.
>
> Regards
> Vikas Srivastava
>
>
> On Thu, Jan 12, 2012 at 1:29 PM, Bejoy Ks <[EMAIL PROTECTED]> wrote:
>
>> Hi Vikas
>>        From the job tracker WebUI, looks like your job is just map only,
>> ie there is no reduce tasks for your job. It is a hive job then if no
>> reduce tasks are required for your query Hive sets the number of reduce
>> tasks to zero at code level. The parameters set using -D at run time on CLI
>> could be overridden at code level. I believe that is happening here.
>>       On the map tasks, there would be constrains taken here like data
>> locality(rack locality etc). May be your data is not uniformly distributed
>> across the cluster or so. Need detailed investigation, can't say in one
>> look.
>>
>> Regards
>> Bejoy.K.S
>>
>>   ------------------------------
>> *From:* hadoop hive <[EMAIL PROTECTED]>
>> *To:* [EMAIL PROTECTED]
>> *Sent:* Thursday, January 12, 2012 12:13 PM
>> *Subject:* Proper utilization of Map and reduce
>>
>> hey all,
>>
>> I have a cluster of 13 nodes, in which i configured map =16 and reduce=10
>> , and mapred.reduce.tasks=120 , replication factor=2
>>
>> but still i wont be able to utilize map and reduce as shown in pic.
>>
>> Thanks in advance.. tell how can i make it utilize full map adn reduce.
>>
>> regards
>> Vikas Srivastava
>>
>>
>>
>
--
"...:::Aniket:::... Quetzalco@tl"
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB