Eric Yang
2011-09-11, 18:04
Todd Lipcon
2011-09-11, 22:13
Gary Helmling
2011-09-11, 22:57
Todd Lipcon
2011-09-11, 23:01
Eric Yang
2011-09-12, 01:00
Gary Helmling
2011-09-12, 22:58
Eric Yang
2011-09-13, 02:35
-
Secure Hadoop and non-secure HBaseEric Yang 2011-09-11, 18:04
Hi all,
Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to work with HBase 0.90.x. However, secure Hadoop and HBase would work until kerberos token expires. There is currently no code that renews kerberos token in HBase. Hence, it is possible to add a cron job to periodically renew the HBase user token to keep the system running. What does the community think about having a setup script for cron job as part of HBase upcoming minor release, and fix the token renewal in HBase code for the next major version. On the other hand, would the community accept the token renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what is the time line for 0.90.5?
regards,
Eric
Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to work with HBase 0.90.x. However, secure Hadoop and HBase would work until kerberos token expires. There is currently no code that renews kerberos token in HBase. Hence, it is possible to add a cron job to periodically renew the HBase user token to keep the system running. What does the community think about having a setup script for cron job as part of HBase upcoming minor release, and fix the token renewal in HBase code for the next major version. On the other hand, would the community accept the token renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what is the time line for 0.90.5?
regards,
Eric
-
Re: Secure Hadoop and non-secure HBaseTodd Lipcon 2011-09-11, 22:13
Hi Eric,
Could you please explain more fully what you mean by this? The daemons
generally run using keytabs, not user credentials, and thus shouldn't
need the explicit TGT Renewer, right?
-Todd
On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to work with HBase 0.90.x. However, secure Hadoop and HBase would work until kerberos token expires. There is currently no code that renews kerberos token in HBase. Hence, it is possible to add a cron job to periodically renew the HBase user token to keep the system running. What does the community think about having a setup script for cron job as part of HBase upcoming minor release, and fix the token renewal in HBase code for the next major version. On the other hand, would the community accept the token renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what is the time line for 0.90.5?
>
> regards,
> Eric
--
Todd Lipcon
Software Engineer, Cloudera
Could you please explain more fully what you mean by this? The daemons
generally run using keytabs, not user credentials, and thus shouldn't
need the explicit TGT Renewer, right?
-Todd
On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to work with HBase 0.90.x. However, secure Hadoop and HBase would work until kerberos token expires. There is currently no code that renews kerberos token in HBase. Hence, it is possible to add a cron job to periodically renew the HBase user token to keep the system running. What does the community think about having a setup script for cron job as part of HBase upcoming minor release, and fix the token renewal in HBase code for the next major version. On the other hand, would the community accept the token renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what is the time line for 0.90.5?
>
> regards,
> Eric
--
Todd Lipcon
Software Engineer, Cloudera
-
Re: Secure Hadoop and non-secure HBaseGary Helmling 2011-09-11, 22:57
Hi Eric,
If you configure
hbase.master.keytab.file
hbase.master.kerberos.principal
hbase.regionserver.keytab.file
hbase.regionserver.kerberos.principal
in your hbase-site.xml, then the master and region server processes should
login from the keytab files on startup, as Todd mentions. It's also my
understanding that they don't need a renewal thread in that case. The RPC
client just tries a relogin from the keytab in the case of a connection
error.
Can you describe a bit more what you're seeing so that we can understand the
context?
Gary
On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Hi Eric,
>
> Could you please explain more fully what you mean by this? The daemons
> generally run using keytabs, not user credentials, and thus shouldn't
> need the explicit TGT Renewer, right?
>
> -Todd
>
> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> > Hi all,
> >
> > Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
> work with HBase 0.90.x. However, secure Hadoop and HBase would work until
> kerberos token expires. There is currently no code that renews kerberos
> token in HBase. Hence, it is possible to add a cron job to periodically
> renew the HBase user token to keep the system running. What does the
> community think about having a setup script for cron job as part of HBase
> upcoming minor release, and fix the token renewal in HBase code for the next
> major version. On the other hand, would the community accept the token
> renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what
> is the time line for 0.90.5?
> >
> > regards,
> > Eric
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
If you configure
hbase.master.keytab.file
hbase.master.kerberos.principal
hbase.regionserver.keytab.file
hbase.regionserver.kerberos.principal
in your hbase-site.xml, then the master and region server processes should
login from the keytab files on startup, as Todd mentions. It's also my
understanding that they don't need a renewal thread in that case. The RPC
client just tries a relogin from the keytab in the case of a connection
error.
Can you describe a bit more what you're seeing so that we can understand the
context?
Gary
On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Hi Eric,
>
> Could you please explain more fully what you mean by this? The daemons
> generally run using keytabs, not user credentials, and thus shouldn't
> need the explicit TGT Renewer, right?
>
> -Todd
>
> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> > Hi all,
> >
> > Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
> work with HBase 0.90.x. However, secure Hadoop and HBase would work until
> kerberos token expires. There is currently no code that renews kerberos
> token in HBase. Hence, it is possible to add a cron job to periodically
> renew the HBase user token to keep the system running. What does the
> community think about having a setup script for cron job as part of HBase
> upcoming minor release, and fix the token renewal in HBase code for the next
> major version. On the other hand, would the community accept the token
> renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what
> is the time line for 0.90.5?
> >
> > regards,
> > Eric
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
-
Re: Secure Hadoop and non-secure HBaseTodd Lipcon 2011-09-11, 23:01
On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to work with HBase 0.90.x.
A slight correction: I think you mean to say: "Hortonworks employees
have revisited their decision to -1 the work that Cloudera, Facebook,
and Stumbleupon did 18 months ago to allow Hadoop 0.20 to work with
HBase. And thus it has been merged into 0.20.205." :)
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
> Hi all,
>
> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to work with HBase 0.90.x.
A slight correction: I think you mean to say: "Hortonworks employees
have revisited their decision to -1 the work that Cloudera, Facebook,
and Stumbleupon did 18 months ago to allow Hadoop 0.20 to work with
HBase. And thus it has been merged into 0.20.205." :)
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
-
Re: Secure Hadoop and non-secure HBaseEric Yang 2011-09-12, 01:00
Hi Gary,
This is exactly what I missed. Without those settings in my hbase-site.xml, I saw these messages in regional server log file:
2011-09-09 17:28:14,172 WARN org.apache.hadoop.ipc.Client: Couldn't setup connection for [EMAIL PROTECTED] tonn/[EMAIL PROTECTED]
2011-09-09 17:28:14,172 WARN org.apache.hadoop.hdfs.DFSClient: Problem renewing lease for DFSClient_hb_rs_hrt8n38.cc1.ygridcore.net,60020,1315520835373_1315520835752
java.io.IOException: Call to hrt8n37.cc1.ygridcore.net/98.137.233.217:8020 failed on local exception: java.io.IOException: Couldn't setup connection [EMAIL PROTECTED] to nn/[EMAIL PROTECTED]
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1104)
at org.apache.hadoop.ipc.Client.call(Client.java:1072)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy9.renewLease(Unknown Source)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy9.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1176)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1188)
at java.lang.Thread.run(Thread.java:619)
The region server can respond query but data is not committed to HDFS. I manually run kinit on the command line, and the log shows:
2011-09-09 17:28:50,466 INFO org.apache.hadoop.security.UserGroupInformation: Initiating logout for [EMAIL PROTECTED]
2011-09-09 17:28:50,466 INFO org.apache.hadoop.security.UserGroupInformation: Initiating re-login for [EMAIL PROTECTED]
I will put the keytab and kerberos.principal in the configuration file to verify token renewal in HBase. Thank you.
regards,
Eric
On Sep 11, 2011, at 3:57 PM, Gary Helmling wrote:
> Hi Eric,
>
> If you configure
>
> hbase.master.keytab.file
> hbase.master.kerberos.principal
> hbase.regionserver.keytab.file
> hbase.regionserver.kerberos.principal
>
> in your hbase-site.xml, then the master and region server processes should
> login from the keytab files on startup, as Todd mentions. It's also my
> understanding that they don't need a renewal thread in that case. The RPC
> client just tries a relogin from the keytab in the case of a connection
> error.
>
> Can you describe a bit more what you're seeing so that we can understand the
> context?
>
> Gary
>
>
> On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> Hi Eric,
>>
>> Could you please explain more fully what you mean by this? The daemons
>> generally run using keytabs, not user credentials, and thus shouldn't
>> need the explicit TGT Renewer, right?
>>
>> -Todd
>>
>> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>> Hi all,
>>>
>>> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
>> work with HBase 0.90.x. However, secure Hadoop and HBase would work until
>> kerberos token expires. There is currently no code that renews kerberos
>> token in HBase. Hence, it is possible to add a cron job to periodically
>> renew the HBase user token to keep the system running. What does the
>> community think about having a setup script for cron job as part of HBase
>> upcoming minor release, and fix the token renewal in HBase code for the next
>> major version. On the other hand, would the community accept the token
>> renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what
>> is the time line for 0.90.5?
>>>
>>> regards,
>>> Eric
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
This is exactly what I missed. Without those settings in my hbase-site.xml, I saw these messages in regional server log file:
2011-09-09 17:28:14,172 WARN org.apache.hadoop.ipc.Client: Couldn't setup connection for [EMAIL PROTECTED] tonn/[EMAIL PROTECTED]
2011-09-09 17:28:14,172 WARN org.apache.hadoop.hdfs.DFSClient: Problem renewing lease for DFSClient_hb_rs_hrt8n38.cc1.ygridcore.net,60020,1315520835373_1315520835752
java.io.IOException: Call to hrt8n37.cc1.ygridcore.net/98.137.233.217:8020 failed on local exception: java.io.IOException: Couldn't setup connection [EMAIL PROTECTED] to nn/[EMAIL PROTECTED]
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1104)
at org.apache.hadoop.ipc.Client.call(Client.java:1072)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy9.renewLease(Unknown Source)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy9.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1176)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1188)
at java.lang.Thread.run(Thread.java:619)
The region server can respond query but data is not committed to HDFS. I manually run kinit on the command line, and the log shows:
2011-09-09 17:28:50,466 INFO org.apache.hadoop.security.UserGroupInformation: Initiating logout for [EMAIL PROTECTED]
2011-09-09 17:28:50,466 INFO org.apache.hadoop.security.UserGroupInformation: Initiating re-login for [EMAIL PROTECTED]
I will put the keytab and kerberos.principal in the configuration file to verify token renewal in HBase. Thank you.
regards,
Eric
On Sep 11, 2011, at 3:57 PM, Gary Helmling wrote:
> Hi Eric,
>
> If you configure
>
> hbase.master.keytab.file
> hbase.master.kerberos.principal
> hbase.regionserver.keytab.file
> hbase.regionserver.kerberos.principal
>
> in your hbase-site.xml, then the master and region server processes should
> login from the keytab files on startup, as Todd mentions. It's also my
> understanding that they don't need a renewal thread in that case. The RPC
> client just tries a relogin from the keytab in the case of a connection
> error.
>
> Can you describe a bit more what you're seeing so that we can understand the
> context?
>
> Gary
>
>
> On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> Hi Eric,
>>
>> Could you please explain more fully what you mean by this? The daemons
>> generally run using keytabs, not user credentials, and thus shouldn't
>> need the explicit TGT Renewer, right?
>>
>> -Todd
>>
>> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>> Hi all,
>>>
>>> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
>> work with HBase 0.90.x. However, secure Hadoop and HBase would work until
>> kerberos token expires. There is currently no code that renews kerberos
>> token in HBase. Hence, it is possible to add a cron job to periodically
>> renew the HBase user token to keep the system running. What does the
>> community think about having a setup script for cron job as part of HBase
>> upcoming minor release, and fix the token renewal in HBase code for the next
>> major version. On the other hand, would the community accept the token
>> renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what
>> is the time line for 0.90.5?
>>>
>>> regards,
>>> Eric
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
-
Re: Secure Hadoop and non-secure HBaseGary Helmling 2011-09-12, 22:58
Hi Eric,
Glad that worked for you. We need to get some information on the security
configuration into the HBase online docs. Let us know if you run into
further problems.
Gary
On Sun, Sep 11, 2011 at 6:00 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
> Hi Gary,
>
> This is exactly what I missed. Without those settings in my
> hbase-site.xml, I saw these messages in regional server log file:
>
> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.ipc.Client: Couldn't setup
> connection for [EMAIL PROTECTED] tonn/
> [EMAIL PROTECTED]
> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.hdfs.DFSClient: Problem
> renewing lease for DFSClient_hb_rs_hrt8n38.cc1.ygridcore.net
> ,60020,1315520835373_1315520835752
> java.io.IOException: Call to hrt8n37.cc1.ygridcore.net/98.137.233.217:8020failed on local exception: java.io.IOException: Couldn't setup connection
> [EMAIL PROTECTED] to nn/[EMAIL PROTECTED]
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1104)
> at org.apache.hadoop.ipc.Client.call(Client.java:1072)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at $Proxy9.renewLease(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy9.renewLease(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1176)
> at
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1188)
> at java.lang.Thread.run(Thread.java:619)
>
> The region server can respond query but data is not committed to HDFS. I
> manually run kinit on the command line, and the log shows:
>
> 2011-09-09 17:28:50,466 INFO
> org.apache.hadoop.security.UserGroupInformation: Initiating logout for
> [EMAIL PROTECTED]
> 2011-09-09 17:28:50,466 INFO
> org.apache.hadoop.security.UserGroupInformation: Initiating re-login for
> [EMAIL PROTECTED]
>
> I will put the keytab and kerberos.principal in the configuration file to
> verify token renewal in HBase. Thank you.
>
> regards,
> Eric
>
> On Sep 11, 2011, at 3:57 PM, Gary Helmling wrote:
>
> > Hi Eric,
> >
> > If you configure
> >
> > hbase.master.keytab.file
> > hbase.master.kerberos.principal
> > hbase.regionserver.keytab.file
> > hbase.regionserver.kerberos.principal
> >
> > in your hbase-site.xml, then the master and region server processes
> should
> > login from the keytab files on startup, as Todd mentions. It's also my
> > understanding that they don't need a renewal thread in that case. The
> RPC
> > client just tries a relogin from the keytab in the case of a connection
> > error.
> >
> > Can you describe a bit more what you're seeing so that we can understand
> the
> > context?
> >
> > Gary
> >
> >
> > On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Eric,
> >>
> >> Could you please explain more fully what you mean by this? The daemons
> >> generally run using keytabs, not user credentials, and thus shouldn't
> >> need the explicit TGT Renewer, right?
> >>
> >> -Todd
> >>
> >> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> >>> Hi all,
> >>>
> >>> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
> >> work with HBase 0.90.x. However, secure Hadoop and HBase would work
> until
> >> kerberos token expires. There is currently no code that renews kerberos
> >> token in HBase. Hence, it is possible to add a cron job to periodically
> >> renew the HBase user token to keep the system running. What does the
> >> community think about having a setup script for cron job as part of
Glad that worked for you. We need to get some information on the security
configuration into the HBase online docs. Let us know if you run into
further problems.
Gary
On Sun, Sep 11, 2011 at 6:00 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
> Hi Gary,
>
> This is exactly what I missed. Without those settings in my
> hbase-site.xml, I saw these messages in regional server log file:
>
> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.ipc.Client: Couldn't setup
> connection for [EMAIL PROTECTED] tonn/
> [EMAIL PROTECTED]
> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.hdfs.DFSClient: Problem
> renewing lease for DFSClient_hb_rs_hrt8n38.cc1.ygridcore.net
> ,60020,1315520835373_1315520835752
> java.io.IOException: Call to hrt8n37.cc1.ygridcore.net/98.137.233.217:8020failed on local exception: java.io.IOException: Couldn't setup connection
> [EMAIL PROTECTED] to nn/[EMAIL PROTECTED]
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1104)
> at org.apache.hadoop.ipc.Client.call(Client.java:1072)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at $Proxy9.renewLease(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy9.renewLease(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1176)
> at
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1188)
> at java.lang.Thread.run(Thread.java:619)
>
> The region server can respond query but data is not committed to HDFS. I
> manually run kinit on the command line, and the log shows:
>
> 2011-09-09 17:28:50,466 INFO
> org.apache.hadoop.security.UserGroupInformation: Initiating logout for
> [EMAIL PROTECTED]
> 2011-09-09 17:28:50,466 INFO
> org.apache.hadoop.security.UserGroupInformation: Initiating re-login for
> [EMAIL PROTECTED]
>
> I will put the keytab and kerberos.principal in the configuration file to
> verify token renewal in HBase. Thank you.
>
> regards,
> Eric
>
> On Sep 11, 2011, at 3:57 PM, Gary Helmling wrote:
>
> > Hi Eric,
> >
> > If you configure
> >
> > hbase.master.keytab.file
> > hbase.master.kerberos.principal
> > hbase.regionserver.keytab.file
> > hbase.regionserver.kerberos.principal
> >
> > in your hbase-site.xml, then the master and region server processes
> should
> > login from the keytab files on startup, as Todd mentions. It's also my
> > understanding that they don't need a renewal thread in that case. The
> RPC
> > client just tries a relogin from the keytab in the case of a connection
> > error.
> >
> > Can you describe a bit more what you're seeing so that we can understand
> the
> > context?
> >
> > Gary
> >
> >
> > On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Eric,
> >>
> >> Could you please explain more fully what you mean by this? The daemons
> >> generally run using keytabs, not user credentials, and thus shouldn't
> >> need the explicit TGT Renewer, right?
> >>
> >> -Todd
> >>
> >> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> >>> Hi all,
> >>>
> >>> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
> >> work with HBase 0.90.x. However, secure Hadoop and HBase would work
> until
> >> kerberos token expires. There is currently no code that renews kerberos
> >> token in HBase. Hence, it is possible to add a cron job to periodically
> >> renew the HBase user token to keep the system running. What does the
> >> community think about having a setup script for cron job as part of
-
Re: Secure Hadoop and non-secure HBaseEric Yang 2011-09-13, 02:35
I verified the configuration are working with secure append patch for 0.20.205.0. The parameters are also documented in HBase Configuration book. Thank you.
regards,
Eric
On Sep 12, 2011, at 3:58 PM, Gary Helmling wrote:
> Hi Eric,
>
> Glad that worked for you. We need to get some information on the security
> configuration into the HBase online docs. Let us know if you run into
> further problems.
>
> Gary
>
>
> On Sun, Sep 11, 2011 at 6:00 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> Hi Gary,
>>
>> This is exactly what I missed. Without those settings in my
>> hbase-site.xml, I saw these messages in regional server log file:
>>
>> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.ipc.Client: Couldn't setup
>> connection for [EMAIL PROTECTED] tonn/
>> [EMAIL PROTECTED]
>> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.hdfs.DFSClient: Problem
>> renewing lease for DFSClient_hb_rs_hrt8n38.cc1.ygridcore.net
>> ,60020,1315520835373_1315520835752
>> java.io.IOException: Call to hrt8n37.cc1.ygridcore.net/98.137.233.217:8020failed on local exception: java.io.IOException: Couldn't setup connection
>> [EMAIL PROTECTED] to nn/[EMAIL PROTECTED]
>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1104)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1072)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>> at $Proxy9.renewLease(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at $Proxy9.renewLease(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1176)
>> at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1188)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> The region server can respond query but data is not committed to HDFS. I
>> manually run kinit on the command line, and the log shows:
>>
>> 2011-09-09 17:28:50,466 INFO
>> org.apache.hadoop.security.UserGroupInformation: Initiating logout for
>> [EMAIL PROTECTED]
>> 2011-09-09 17:28:50,466 INFO
>> org.apache.hadoop.security.UserGroupInformation: Initiating re-login for
>> [EMAIL PROTECTED]
>>
>> I will put the keytab and kerberos.principal in the configuration file to
>> verify token renewal in HBase. Thank you.
>>
>> regards,
>> Eric
>>
>> On Sep 11, 2011, at 3:57 PM, Gary Helmling wrote:
>>
>>> Hi Eric,
>>>
>>> If you configure
>>>
>>> hbase.master.keytab.file
>>> hbase.master.kerberos.principal
>>> hbase.regionserver.keytab.file
>>> hbase.regionserver.kerberos.principal
>>>
>>> in your hbase-site.xml, then the master and region server processes
>> should
>>> login from the keytab files on startup, as Todd mentions. It's also my
>>> understanding that they don't need a renewal thread in that case. The
>> RPC
>>> client just tries a relogin from the keytab in the case of a connection
>>> error.
>>>
>>> Can you describe a bit more what you're seeing so that we can understand
>> the
>>> context?
>>>
>>> Gary
>>>
>>>
>>> On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Eric,
>>>>
>>>> Could you please explain more fully what you mean by this? The daemons
>>>> generally run using keytabs, not user credentials, and thus shouldn't
>>>> need the explicit TGT Renewer, right?
>>>>
>>>> -Todd
>>>>
>>>> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>>>> Hi all,
>>>>>
>>>>> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
>>>> work with HBase 0.90.x. However, secure Hadoop and HBase would work
regards,
Eric
On Sep 12, 2011, at 3:58 PM, Gary Helmling wrote:
> Hi Eric,
>
> Glad that worked for you. We need to get some information on the security
> configuration into the HBase online docs. Let us know if you run into
> further problems.
>
> Gary
>
>
> On Sun, Sep 11, 2011 at 6:00 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> Hi Gary,
>>
>> This is exactly what I missed. Without those settings in my
>> hbase-site.xml, I saw these messages in regional server log file:
>>
>> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.ipc.Client: Couldn't setup
>> connection for [EMAIL PROTECTED] tonn/
>> [EMAIL PROTECTED]
>> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.hdfs.DFSClient: Problem
>> renewing lease for DFSClient_hb_rs_hrt8n38.cc1.ygridcore.net
>> ,60020,1315520835373_1315520835752
>> java.io.IOException: Call to hrt8n37.cc1.ygridcore.net/98.137.233.217:8020failed on local exception: java.io.IOException: Couldn't setup connection
>> [EMAIL PROTECTED] to nn/[EMAIL PROTECTED]
>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1104)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1072)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>> at $Proxy9.renewLease(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at $Proxy9.renewLease(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1176)
>> at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1188)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> The region server can respond query but data is not committed to HDFS. I
>> manually run kinit on the command line, and the log shows:
>>
>> 2011-09-09 17:28:50,466 INFO
>> org.apache.hadoop.security.UserGroupInformation: Initiating logout for
>> [EMAIL PROTECTED]
>> 2011-09-09 17:28:50,466 INFO
>> org.apache.hadoop.security.UserGroupInformation: Initiating re-login for
>> [EMAIL PROTECTED]
>>
>> I will put the keytab and kerberos.principal in the configuration file to
>> verify token renewal in HBase. Thank you.
>>
>> regards,
>> Eric
>>
>> On Sep 11, 2011, at 3:57 PM, Gary Helmling wrote:
>>
>>> Hi Eric,
>>>
>>> If you configure
>>>
>>> hbase.master.keytab.file
>>> hbase.master.kerberos.principal
>>> hbase.regionserver.keytab.file
>>> hbase.regionserver.kerberos.principal
>>>
>>> in your hbase-site.xml, then the master and region server processes
>> should
>>> login from the keytab files on startup, as Todd mentions. It's also my
>>> understanding that they don't need a renewal thread in that case. The
>> RPC
>>> client just tries a relogin from the keytab in the case of a connection
>>> error.
>>>
>>> Can you describe a bit more what you're seeing so that we can understand
>> the
>>> context?
>>>
>>> Gary
>>>
>>>
>>> On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Eric,
>>>>
>>>> Could you please explain more fully what you mean by this? The daemons
>>>> generally run using keytabs, not user credentials, and thus shouldn't
>>>> need the explicit TGT Renewer, right?
>>>>
>>>> -Todd
>>>>
>>>> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>>>> Hi all,
>>>>>
>>>>> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
>>>> work with HBase 0.90.x. However, secure Hadoop and HBase would work
