Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - MiniMRCluster not behaving in hadoop 1.0.4

Copy link to this message
MiniMRCluster not behaving in hadoop 1.0.4
Koert Kuipers 2012-11-10, 07:43
i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the
unit tests used to run fine but are now misbehaving. i do a simple setup in
the unit test like this:

    private static MiniDFSCluster dfsCluster;
    private static FileSystem fs;
    private static MiniMRCluster mrCluster;

    public static void oneTimeSetUp() throws Exception {
        new File("tmp/logs").mkdirs();
        System.setProperty("hadoop.log.dir", "tmp/logs");

        final Configuration conf = new Configuration();
        conf.set("io.skip.checksum.errors", "true");
        dfsCluster = new MiniDFSCluster(conf, 2, true, null);
        fs = dfsCluster.getFileSystem();
        mrCluster = new MiniMRCluster(2,
dfsCluster.getFileSystem().getUri().toString(), 1, null, null, new

however with hadoop 1.0.4 this results in an exception where the jobtracker

12/11/10 02:38:29 ERROR mapred.MiniMRCluster: Job tracker crashed
java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
    at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150)

not sure what this error is about, but i managed to get around it for now
by removing the 2 hadoop apache jasper dependencies which seem to have
something to do with it: jasper-compiler-5.5.12.jar and
jasper-runtime-5.5.12.jar. with this "fix" i get further and my jobs
starts, but now all the tasks fail with messages like this:

attempt_20121110024037623_0001_r_000002_0: Exception in thread "main"
java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child

am i doing something completely wrong, and was i just lucky that this used
to run with CDH3? any ideas? thanks! koert
Steve Loughran 2012-11-10, 10:24
Koert Kuipers 2012-11-10, 19:42
Steve Loughran 2012-11-11, 18:04