Hi list,

I've spent the whole of yesterday debugging a DN to find and fix a trivial bug (HDFS-3059). Difficulty is that we have Kerberos enabled and I didn't know how to emulate that environment locally. So what I did was editing the source, building the jars, deploying them on my NN and a single DN, starting deamons, and looking at my debug statements.

Obviously not the most optimal way. How do you debug deamons with Kerberos enabled? Do you add your development machine to the cluster, give it a principal, and configure the network? Any tips / best practices? I expect we're going to need this more often since we're just bringing up a fair sized multi-tenant production environment (~70 machines) and I guess this won't be the only time we'll want to see what exactly happens.

Thanks,
Evert

Why not just attach a debugger from an IDE (with the right hadoop sources)
to DN JVM?

On Thu, Mar 8, 2012 at 1:16 AM, Evert Lammerts <[EMAIL PROTECTED]>wrote:

> Hi list,
>
> I've spent the whole of yesterday debugging a DN to find and fix a trivial
> bug (HDFS-3059). Difficulty is that we have Kerberos enabled and I didn't
> know how to emulate that environment locally. So what I did was editing the
> source, building the jars, deploying them on my NN and a single DN,
> starting deamons, and looking at my debug statements.
>
> Obviously not the most optimal way. How do you debug deamons with Kerberos
> enabled? Do you add your development machine to the cluster, give it a
> principal, and configure the network? Any tips / best practices? I expect
> we're going to need this more often since we're just bringing up a fair
> sized multi-tenant production environment (~70 machines) and I guess this
> won't be the only time we'll want to see what exactly happens.
>
> Thanks,
> Evert
>

+ common-dev
bcc: general

Moving this to the more appropriate common-dev@

I personally like to run a KDC on my dev box and build/deploy a
pseudo-distributed cluster right there. This lets me make principals at
will, adjust krb5 settings as I please, regenerate keytabs, etc. Setting up
a local MIT KDC is actually quite easy.

--
Aaron T. Myers
Software Engineer, Cloudera

On Thu, Mar 8, 2012 at 4:05 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>wrote:

> Why not just attach a debugger from an IDE (with the right hadoop sources)
> to DN JVM?
>
> On Thu, Mar 8, 2012 at 1:16 AM, Evert Lammerts <[EMAIL PROTECTED]
> >wrote:
>
> > Hi list,
> >
> > I've spent the whole of yesterday debugging a DN to find and fix a
> trivial
> > bug (HDFS-3059). Difficulty is that we have Kerberos enabled and I didn't
> > know how to emulate that environment locally. So what I did was editing
> the
> > source, building the jars, deploying them on my NN and a single DN,
> > starting deamons, and looking at my debug statements.
> >
> > Obviously not the most optimal way. How do you debug deamons with
> Kerberos
> > enabled? Do you add your development machine to the cluster, give it a
> > principal, and configure the network? Any tips / best practices? I expect
> > we're going to need this more often since we're just bringing up a fair
> > sized multi-tenant production environment (~70 machines) and I guess this
> > won't be the only time we'll want to see what exactly happens.
> >
> > Thanks,
> > Evert
> >
>

Hi Evert,
  HADOOP-8078 was recently committed that lets you bring up a kdc in unit
tests and you can test namenode with security on. However, datanodes still
don't come up because they require privileged ports when security is
enabled. But, there is a workaround patch attached to HDFS-3064 which will
allow datanodes to come up as well. Please use same principals for namenode
and all datanodes. The keytabs are also committed as part of HADOOP-8078.

regards

On Thu, Mar 8, 2012 at 1:16 AM, Evert Lammerts <[EMAIL PROTECTED]>wrote:

> Hi list,
>
> I've spent the whole of yesterday debugging a DN to find and fix a trivial
> bug (HDFS-3059). Difficulty is that we have Kerberos enabled and I didn't
> know how to emulate that environment locally. So what I did was editing the
> source, building the jars, deploying them on my NN and a single DN,
> starting deamons, and looking at my debug statements.
>
> Obviously not the most optimal way. How do you debug deamons with Kerberos
> enabled? Do you add your development machine to the cluster, give it a
> principal, and configure the network? Any tips / best practices? I expect
> we're going to need this more often since we're just bringing up a fair
> sized multi-tenant production environment (~70 machines) and I guess this
> won't be the only time we'll want to see what exactly happens.
>
> Thanks,
> Evert
>

--