Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Re: Working with Capacity Scheduler


Copy link to this message
-
Re: Working with Capacity Scheduler
Munna 2013-11-27, 00:02
Hi olivier,

Sorry to again, same error: Job error given below

[mapred@host~]$ hive -e "use databse; select count(*) from table"
Logging initialized using configuration in
jar:file:/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hive/lib/hive-common-0.10.0-cdh4.4.0.jar!/hive-log4j.properties
Hive history
file=/tmp/mapred/hive_job_log_868a2d7a-1df8-4a8a-b09d-4fad10d3f1b4_1410849451.txt
OK
Time taken: 1.616 seconds
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_1385510241043_0001, Tracking URL http://host:8088/proxy/application_1385510241043_0001/
Kill Command /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/bin/hadoop job
-kill job_1385510241043_0001
Ended Job = job_1385510241043_0001 with errors
Error during job, obtaining debugging information...
Job Tracking URL:
http://hpdl-R306-16:8088/proxy/application_1385510241043_0001/
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0:  HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

On Wed, Nov 27, 2013 at 5:01 AM, Olivier Renault
<[EMAIL PROTECTED]>wrote:

> Yes you need to forbid every users on your root queue
>
>   <property>
>       <name>yarn.scheduler.capacity.root.acl_submit_applications</name>
>       <value> </value>
>   </property>
>
>   <property>
>       <name>yarn.scheduler.capacity.root.acl_administer_queue</name>
>       <value> </value>
>   </property>
>
> Olivier
>
>
> On 26 November 2013 23:29, Munna <[EMAIL PROTECTED]> wrote:
>
>> I have configured acls like this:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *<property>  <name>yarn.scheduler.capacity.root.
>> production.acl_submit_applications</name>
>> <value>yarn,mapred</value></property><property>    <value>userb</value>
>> <name>yarn.scheduler.capacity.root.exploration.a.acl_submit_applications</name>
>> </property><property>
>> <name>yarn.scheduler.capacity.root.exploration.b.acl_submit_applications</name>
>> <value>userb</value></property><property>
>> <name>yarn.scheduler.capacity.root.exploration.c.acl_submit_applications</name>
>> <value>userc</value></property>*
>>  Is any thing missing on this?
>>
>> Thanks
>>
>>
>> On Wed, Nov 27, 2013 at 4:31 AM, Olivier Renault <
>> [EMAIL PROTECTED]> wrote:
>>
>>> I don't believe the distro should matter.
>>>
>>> Could you confirm that you've got the following in
>>> capacity-scheduler.xml ? ( The permission are hierarchical, so you need to
>>> restrict the root queue )
>>>   <property>
>>>       <name>yarn.scheduler.capacity.root.acl_submit_applications</name>
>>>       <value> </value>
>>>   </property>
>>>
>>>   <property>
>>>       <name>yarn.scheduler.capacity.root.acl_administer_queue</name>
>>>       <value> </value>
>>>
>>> Thanks,
>>> Olivier
>>>
>>>
>>> On 26 November 2013 22:51, Munna <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Olivier,
>>>>
>>>> "yarn.acl.enable"  is enabled earlier, for your information i am using
>>>> Cloudera Manager to manage the cluster. again same problem :(
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Nov 27, 2013 at 3:37 AM, Olivier Renault <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> Here is a working configuration.
>>>>>
>>>>> In yarn-site.xml, you'll need to enable
>>>>>     <name>yarn.acl.enable</name>
>>>>>     <value>true</value>
>>>>>
>>>>> Thanks,
>>>>> Olivier
>>>>>
>>>>>
>>>>> On 26 November 2013 21:48, Munna <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Can i able to get the solution on Capacity Scheduler ?
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 27, 2013 at 2:24 AM, Munna <[EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> yes... acl's are not enforced.

*Regards*

*Munna*