Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # dev >> Re: validating user IDs


Copy link to this message
-
Re: validating user IDs
-hdfs-dev@
+mapreduce-dev@

Moving to the more-relevant mapreduce-dev.

--
Aaron T. Myers
Software Engineer, Cloudera

On Mon, Jun 11, 2012 at 4:12 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>wrote:

> Colin,
>
> Would be possible using some kind of cmake config magic to set a macro to
> the current OS limit? Even if this means detecting the OS version and
> assuming its default limit.
>
> thx
>
> On Mon, Jun 11, 2012 at 3:57 PM, Colin McCabe <[EMAIL PROTECTED]
> >wrote:
>
> > Hi all,
> >
> > I recently pulled the latest source, and ran a full build.  The
> > command line was this:
> > mvn compile -Pnative
> >
> > I was confronted with this:
> >
> > [INFO] Requested user cmccabe has id 500, which is below the minimum
> > allowed 1000
> > [INFO] FAIL: test-container-executor
> > [INFO] ===============================================> > [INFO] 1 of 1 test failed
> > [INFO] Please report to [EMAIL PROTECTED]
> > [INFO] ===============================================> > [INFO] make[1]: *** [check-TESTS] Error 1
> > [INFO] make[1]: Leaving directory
> >
> >
> `/home/cmccabe/hadoop4/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/container-executor'
> >
> > Needless to say, it didn't do much to improve my mood.  I was even
> > less happy when I discovered that -DskipTests has no effect on native
> > tests (they always run.)  See HADOOP-8480.
> >
> > Unfortunately, it seems like this problem is popping up more and more
> > in our native code.  It first appeared in test-task-controller (see
> > MAPREDUCE-2376) and then later in test-container-executor
> > (HADOOP-8499).  The basic problem seems to be the hardcoded assumption
> > that all user IDs below 1000 are system IDs.
> >
> > It is true that there are configuration files that can be changed to
> > alter the minimum user ID, but unfortunately these configuration files
> > are not used by the unit tests.  So anyone developing on a platform
> > where the user IDs start at 500 is now a second-class citizen, unable
> > to run unit tests.  This includes anyone running Red Hat, MacOS,
> > Fedora, etc.
> >
> > Personally, I can change my user ID.  It's a time-consuming process,
> > because I need to re-uid all files, but I can do it.  This luxury may
> > not be available to everyone, though-- developers who don't have root
> > on their machines, or are using a pre-assigned user ID to connect to
> > NFS come to mind.
> >
> > It's true that we could hack around this with environment variables.
> > It might even be possible to have Maven set these environment
> > variables automatically from the current user ID.  However, the larger
> > question I have here is whether this UID validation scheme even makes
> > any sense.  I have a user named "nobody" whose user ID is 65534.
> > Surely I should not be able to run map-reduce jobs as this user?  Yet,
> > under the current system, I can do exactly that.  The root of the
> > problem seems to be that there is both a default minimum and a default
> > maximum for "automatic" user IDs.  This configuration seems to be
> > stored in /etc/login.defs.
> >
> > On my system, it has:
> > SYSTEM_UID_MIN            100
> > SYSTEM_UID_MAX            499
> > UID_MIN                  500
> > UID_MAX                 60000
> >
> > So that means that anything over 60000 (like nobody) is not considered
> > a valid user ID for regular users.
> > We could potentially read this file (at least on Linux) and get more
> > sensible defaults.
> >
> > I am also curious if we could simply check whether the user we're
> > trying to run the job as has a valid login shell.  System users are
> > almost always set to have a login shell of /bin/false or
> > /sbin/nologin.
> >
> > Thoughts?
> > Colin
> >
>
>
>
> --
> Alejandro
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB