Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Suggest that turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level from ERROR to WARN


Copy link to this message
-
Re: Suggest that turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level from ERROR to WARN
Lets change it in both places.  Please file issues.  Lets try minimize the
freakout incidents running your hbase/hadoop cluster.
Thanks Takeshi,
St.Ack
On Thu, Jan 16, 2014 at 9:57 PM, takeshi <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> Recently we got the error msg "Request is a replay (34) - PROCESS_TGS"
> while we are using the HBase client API to put data into HBase-0.94.16 with
> krb5-1.6.1 enabled. The related msg as follows...
> {code}
> [2014-01-15
>
> 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
> PriviledgedActionException
> as:takeshi_miao@LABcause:javax.security.sasl.SaslException: GSS
> initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15
>
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
> PriviledgedAction
> as:takeshi_miao@LABfrom
> :sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> [2014-01-15
>
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
> Exception encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
>
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
> Initiating logout for takeshi_miao@LAB
> [2014-01-15
>
> 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
> hadoop logout
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
>
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
> Initiating re-login for takeshi_miao@LAB
> [2014-01-15
>
> 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
> hadoop login
> [2014-01-15
>
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
> hadoop login commit
> [2014-01-15
>
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
> using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao,
> UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]:
> 501, UnixNumericGroupPrincipal [Supplementary Group]: 502, takeshi_miao@LAB
> ]
> {code}
>
> At the beginning, we were worry about the "data loss" occurring while we
> found the "Request is a replay (34) - PROCESS_TGS" (especially it is the
> ERROR level) in log, but after code study, this is basically *NOT* a data
> loss issue due to HBase client API would try 5 times internally
> (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of
> retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all
> failed thread), The HTable API would also throw IOEcxeption to client code
> if any thread still fail after these retries.
>
> Based on HBase users' viewpoint as us, we think this is better to change
> the logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had been