Oozie question

I'm trying to run an Oozie workflow (sqoop action) from the Hue console and it fails every time.  No exception in the oozie log but I see this in the Job Tracker log file.

Two primary issues seem to be

1.   Client mapred tries to renew a token with renewer specified as mr token

And
2.   Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN

Any ideas how to get past this?

Full Stacktrace:

2013-01-29 17:11:28,860 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Creating password for identifier: owner=hdfs, renewer=mr token, realUser=oozie, issueDate=1359501088860, maxDate=1360105888860, sequenceNumber=75, masterKeyId=8
2013-01-29 17:11:28,871 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Creating password for identifier: owner=hdfs, renewer=mr token, realUser=oozie, issueDate=1359501088871, maxDate=1360105888871, sequenceNumber=76, masterKeyId=8
2013-01-29 17:11:29,202 INFO org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: registering token for renewal for service =10.204.12.62:8021 and jobID = job_201301231648_0029
2013-01-29 17:11:29,211 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Token renewal requested for identifier: owner=hdfs, renewer=mr token, realUser=oozie, issueDate=1359501088871, maxDate=1360105888871, sequenceNumber=76, masterKeyId=8
2013-01-29 17:11:29,211 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:mapred (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as mr token
2013-01-29 17:11:29,211 WARN org.apache.hadoop.security.token.Token: Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN
2013-01-29 17:11:29,211 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 8021, call renewDelegationToken(Kind: MAPREDUCE_DELEGATION_TOKEN, Service: 10.204.12.62:8021, Ident: 00 04 68 64 66 73 08 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65 8a 01 3c 88 94 58 67 8a 01 3c ac a0 dc 67 4c 08), rpc version=2, client version=28, methodsFingerPrint=1830206421 from 10.204.12.62:9706: error: org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as mr token
org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as mr token
          at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:274)
          at org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3738)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
          at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
          at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
          at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
          at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
2013-01-29 17:11:29,212 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:mapred (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Client mapred tries to renew a token with renewer specified as mr token
          at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:274)
          at org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3738)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
          at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
          at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
          at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
          at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)

2013-01-29 17:11:29,212 WARN org.apache.hadoop.security.token.Token: Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN
2013-01-29 17:11:29,212 ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: Exception renewing tokenKind: MAPREDUCE_DELEGATION_TOKEN, Service: 10.204.12.62:8021, Ident: 00 04 68 64 66 73 08 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65 8a 01 3c 88 94 58 67 8a 01 3c ac a0 dc 67 4c 08. Not rescheduled
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Client mapred tries to renew a token with renewer specified as mr token
          at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:274)
          at org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker

The job that Oozie launches for your action, which you are observing
is failing, does its own logs (task logs) show any errors?

On Wed, Jan 30, 2013 at 4:59 AM, Corbett Martin <[EMAIL PROTECTED]> wrote:
> Oozie question
>
>
>
> I’m trying to run an Oozie workflow (sqoop action) from the Hue console and
> it fails every time.  No exception in the oozie log but I see this in the
> Job Tracker log file.
>
>
>
> Two primary issues seem to be
>
> 1.   Client mapred tries to renew a token with renewer specified as mr token
>
>
>
> And
>
>
>
> 2.   Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN
>
>
>
> Any ideas how to get past this?
>
>
>
> Full Stacktrace:
>
>
>
> 2013-01-29 17:11:28,860 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Creating password for identifier: owner=hdfs, renewer=mr token,
> realUser=oozie, issueDate=1359501088860, maxDate=1360105888860,
> sequenceNumber=75, masterKeyId=8
>
> 2013-01-29 17:11:28,871 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Creating password for identifier: owner=hdfs, renewer=mr token,
> realUser=oozie, issueDate=1359501088871, maxDate=1360105888871,
> sequenceNumber=76, masterKeyId=8
>
> 2013-01-29 17:11:29,202 INFO
> org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal:
> registering token for renewal for service =10.204.12.62:8021 and jobID > job_201301231648_0029
>
> 2013-01-29 17:11:29,211 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Token renewal requested for identifier: owner=hdfs, renewer=mr token,
> realUser=oozie, issueDate=1359501088871, maxDate=1360105888871,
> sequenceNumber=76, masterKeyId=8
>
> 2013-01-29 17:11:29,211 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:mapred (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Client mapred tries
> to renew a token with renewer specified as mr token
>
> 2013-01-29 17:11:29,211 WARN org.apache.hadoop.security.token.Token: Cannot
> find class for token kind MAPREDUCE_DELEGATION_TOKEN
>
> 2013-01-29 17:11:29,211 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 9 on 8021, call renewDelegationToken(Kind:
> MAPREDUCE_DELEGATION_TOKEN, Service: 10.204.12.62:8021, Ident: 00 04 68 64
> 66 73 08 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65 8a 01 3c 88 94 58 67 8a
> 01 3c ac a0 dc 67 4c 08), rpc version=2, client version=28,
> methodsFingerPrint=1830206421 from 10.204.12.62:9706: error:
> org.apache.hadoop.security.AccessControlException: Client mapred tries to
> renew a token with renewer specified as mr token
>
> org.apache.hadoop.security.AccessControlException: Client mapred tries to
> renew a token with renewer specified as mr token
>
>           at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:274)
>
>           at
> org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3738)
>
>           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>           at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
>           at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>           at java.lang.reflect.Method.invoke(Method.java:597)
>
>           at
> org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
>
>           at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
>
>           at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
>
>           at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
>
>           at java.security.AccessController.doPrivileged(Native Method)
>
>           at javax.security.auth.Subject.doAs(Subject.java:396)
>
>           at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
>
>           at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)

Harsh J

Cobert,

[Moving thread to user@oozie.a.o, BCCing common-user@hadoop.a.o]

* What version of Oozie are you using?
* Is the cluster a secure setup (Kerberos enabled)?
* Would you mind posting the complete launcher logs?

Thx

On Wed, Jan 30, 2013 at 6:14 AM, Corbett Martin <[EMAIL PROTECTED]> wrote:

> Thanks for the tip.
>
> The sqoop command listed in the stdout log file is:
>                     sqoop
>                     import
>                     --driver
>                     org.apache.derby.jdbc.ClientDriver
>                     --connect
>                     jdbc:derby://test-server:1527/mondb
>                     --username
>                     monuser
>                     --password
>                     xxxxxxxxx
>                     --table
>                     MONITOR
>                     --split-by
>                     request_id
>                     --target-dir
>                     /mon/import
>                     --append
>                     --incremental
>                     append
>                     --check-column
>                     request_id
>                     --last-value
>                     2000000
>
> The following information is from the sterr and stdout log files.
>
> /var/log/hadoop-0.20-mapreduce/userlogs/job_201301231648_0029/attempt_201301231648_0029_m_000000_0
> # more stderr
> No such sqoop tool: sqoop. See 'sqoop help'.
> Intercepting System.exit(1)
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
>
>
> /var/log/hadoop-0.20-mapreduce/userlogs/job_201301231648_0029/attempt_201301231648_0029_m_000000_0
> # more stdout
> ...
> ...
> ...
> >>> Invoking Sqoop command line now >>>
>
> 1598 [main] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has
> not been set in the environment. Cannot check for additional configuration.
> Intercepting System.exit(1)
>
> <<< Invocation of Main class completed <<<
>
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
>
> Oozie Launcher failed, finishing Hadoop job gracefully
>
> Oozie Launcher ends
>
>
> ~Corbett Martin
> Software Architect
> AbsoluteAR Accounts Receivable Services - An NHIN Solution
>
> -----Original Message-----
> From: Harsh J [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, January 29, 2013 10:11 PM
> To: <[EMAIL PROTECTED]>
> Subject: Re: Oozie workflow error - renewing token issue
>
> The job that Oozie launches for your action, which you are observing is
> failing, does its own logs (task logs) show any errors?
>
> On Wed, Jan 30, 2013 at 4:59 AM, Corbett Martin <[EMAIL PROTECTED]> wrote:
> > Oozie question
> >
> >
> >
> > I'm trying to run an Oozie workflow (sqoop action) from the Hue
> > console and it fails every time.  No exception in the oozie log but I
> > see this in the Job Tracker log file.
> >
> >
> >
> > Two primary issues seem to be
> >
> > 1.   Client mapred tries to renew a token with renewer specified as mr
> token
> >
> >
> >
> > And
> >
> >
> >
> > 2.   Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN
> >
> >
> >
> > Any ideas how to get past this?
> >
> >
> >
> > Full Stacktrace:
> >
> >
> >
> > 2013-01-29 17:11:28,860 INFO
> >
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> > Creating password for identifier: owner=hdfs, renewer=mr token,
> > realUser=oozie, issueDate=1359501088860, maxDate=1360105888860,
> > sequenceNumber=75, masterKeyId=8
> >
> > 2013-01-29 17:11:28,871 INFO
> >
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> > Creating password for identifier: owner=hdfs, renewer=mr token,
> > realUser=oozie, issueDate=1359501088871, maxDate=1360105888871,
> > sequenceNumber=76, masterKeyId=8
> >
> > 2013-01-29 17:11:29,202 INFO
> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal:
> > registering token for renewal for service =10.204.12.62:8021 and jobID
> > > > job_201301231648_0029

Alejandro