Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - testHDFSConf.xml


Copy link to this message
-
Re: testHDFSConf.xml
Jay Vyas 2013-04-11, 13:14
Thats great. Yes this is certainly something we could benefit from.

But the "trunk" github appears to only have
"./src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml" in the source tree.

Also, I don't see it in branch-0.22:

# git checkout branch-0.22
Branch branch-0.22 set up to track remote branch branch-0.22 from origin.
Switched to a new branch 'branch-0.22'
# find ./ -name *Conf.xml
./src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml
Maybe I'm missing something?
On Thu, Apr 11, 2013 at 1:28 AM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:

> I have split CLI test infrastructure into hierarchical pieces that allow to
> have different configurations for different components. E.g. you can have
> one
> for YARN that would exist independently of HDFS, etc. The change has been
> in
> since like 0.22 and committed to 0.203.x as well IIRC, hence should be
> usable
> across the Hadoop versions. May be it something you would benefit from.
>
> Cos
>
> On Wed, Apr 10, 2013 at 10:43AM, Colin McCabe wrote:
> > On Wed, Apr 10, 2013 at 10:16 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> >
> > > Hello HDFS brethren !
> > >
> > > I've noticed that the testHDFSConf.xml has alot of references to
> > > supergroup.
> > >
> > >
> > >
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
> > >
> > > 1) I wonder why this is hardcoded in the testHDFSConf.xml
> > >
> > >
> > "supergroup" is the default supergroup in HDFS.  Check
> DFSConfigKeys.java:
> >
> >   public static final String  DFS_PERMISSIONS_SUPERUSERGROUP_KEY > > "dfs.permissions.superusergroup";
> >   public static final String  DFS_PERMISSIONS_SUPERUSERGROUP_DEFAULT > > "supergroup";
> >
> > It seems fine to use "supergroup" in a test.  after all, we do control
> the
> > configuration we pass into the test.
> >
> >
> > > 2) Also, Im wondering if there are any good ideas for
> extending/modifying
> > > this file for a extention of the FileSystem implementation.
> > >
> > >
> > It would be interesting to think about pulling the non-hdfs-specific
> > components of TestHDFSCLI into another test; perhaps one in common.
> >  Theoretically, what we print on the console should be really similar, no
> > matter whether HDFS or some other filesystem is being used.  In practice,
> > there may be some differences, however...
> >
> > I find it a little bit challenging to modify TestHDFSCLI because the test
> > is really long and executes as a single unit.  Breaking it down into
> > multiple units would probably be another good improvement, at least in my
> > opinion.
> >
> > best,
> > Colin
> >
> >
> > Right  now im doing some global find replace statements - but was
> thinking
> > > that maybe parameterizing the file would be a good JIRA - so that
> people
> > > could use this as a base test for FileSystem implementations
> > >
> > > Depending on feedback im certainly willing to submit and put in a first
> > > pass at a more modular version of this file.
> > >
> > > Its in many ways a very generalizable component of the hdfs trunk.
> > >
> > > Thanks!
> > > --
> > > Jay Vyas
> > > http://jayunit100.blogspot.com
> > >
>

--
Jay Vyas
http://jayunit100.blogspot.com