Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Client mapred tries to renew a token with renewer specified as nobody


+
Rainer Toebbicke 2013-12-02, 16:25
+
Rainer Toebbicke 2013-12-04, 08:35
Copy link to this message
-
Re: Client mapred tries to renew a token with renewer specified as nobody
Daryn Sharp 2013-12-09, 14:39
We encountered the same issue in yarn's RM so I made the RM recognize it's own tokens and renew them regardless of the renewer.  In part because older versions of oozie hardcode the renewer as "mrtoken".  I thought the change made it back into 1.x JT but I guess not.

I agree that the conversion to short name is occurring on the "wrong side of the fence".  Auth_to_local is a function of the server, not the client.  Please file a jira.

Dfsclient also shouldn't be repeatedly retrying when the connection fails with InvalidToken.  Hopefully the client didn't try more than 45 times (to clarify your "gazillion" from an earlier message).   Some exceptions are transient, but an invalid token is game over because tokens don't ever become valid again.  HA used to rely on InvalidToken to trigger retries in case the standby is in the processing of catching up on edits, but that's been converted to a RetriableException during a transition to active.   Please file another jira.

I would also expect  _HOST to expand correctly since a client may need to submit to multiple JTs and not want to hassle with using different configs.  Would you like to file another jira? :)

Daryn

On Dec 9, 2013, at 3:13 AM, Rainer Toebbicke <[EMAIL PROTECTED]> wrote:

>
> Le 5 déc. 2013 à 05:30, Vinod Kumar Vavilapalli a écrit :
>
>>
>> It is clearly mentioning that the renewer is wrong (renewer marked is 'nobody' but mapred is trying to renew the token), you may want to check this.
>>
>> Thanks,
>> +Vinod
>>
>> On Dec 2, 2013, at 8:25 AM, Rainer Toebbicke wrote:
>>
>>> 2013-12-02 15:57:08,541 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:mapred/[EMAIL PROTECTED] (auth:KERBEROS) cause:org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as nobody
>>
>
> Thanks, I had already guessed so far and finally the problem turned out a subtle bug in the Cloudera "Configuring MRv1 Security" instructions: they recommend to specify mapreduce.jobtracker.kerberos.principal as mapred/[EMAIL PROTECTED] in mapred-site.xml. That won't work.
>
> It confuses TokenCache.obtainTokensForNamenodesInternal() when it tries to obtain mapred's HadoopKerberosName.getShortname(): it finds the untranslated _HOST which a correctly configured hadoop.security.auth_to_local would not recognize as valid cluster node. That's where "nobody" comes in.
>
> Solution: mapreduce.jobtracker.kerberos.principal should really be full a fully resolved mapred/[EMAIL PROTECTED] and thus would usually pass the hadoop.security.auth_to_local rules.
>
> One could argue that TokenCache.obtainTokensForNamenodesInternal() actually uses getShortname() from the wrong side of the fence but the little change fixes the problem.
>
> Thanks, Rainer