Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - MiniMRCluster not behaving in hadoop 1.0.4


Copy link to this message
-
MiniMRCluster not behaving in hadoop 1.0.4
Koert Kuipers 2012-11-10, 07:43
i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the
unit tests used to run fine but are now misbehaving. i do a simple setup in
the unit test like this:

    private static MiniDFSCluster dfsCluster;
    private static FileSystem fs;
    private static MiniMRCluster mrCluster;

    @BeforeClass
    public static void oneTimeSetUp() throws Exception {
        new File("tmp/logs").mkdirs();
        System.setProperty("hadoop.log.dir", "tmp/logs");
        System.setProperty("javax.xml.parsers.SAXParserFactory",

"com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl");
        final Configuration conf = new Configuration();
        conf.set("io.skip.checksum.errors", "true");
        dfsCluster = new MiniDFSCluster(conf, 2, true, null);
        fs = dfsCluster.getFileSystem();
        mrCluster = new MiniMRCluster(2,
dfsCluster.getFileSystem().getUri().toString(), 1, null, null, new
org.apache.hadoop.mapred.JobConf(conf));
    }

however with hadoop 1.0.4 this results in an exception where the jobtracker
dies:

12/11/10 02:38:29 ERROR mapred.MiniMRCluster: Job tracker crashed
java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
    at
org.apache.jasper.compiler.JspRuntimeContext.<init>(JspRuntimeContext.java:197)
    at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150)
    at
org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)

not sure what this error is about, but i managed to get around it for now
by removing the 2 hadoop apache jasper dependencies which seem to have
something to do with it: jasper-compiler-5.5.12.jar and
jasper-runtime-5.5.12.jar. with this "fix" i get further and my jobs
starts, but now all the tasks fail with messages like this:

attempt_20121110024037623_0001_r_000002_0: Exception in thread "main"
java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child

am i doing something completely wrong, and was i just lucky that this used
to run with CDH3? any ideas? thanks! koert
+
Steve Loughran 2012-11-10, 10:24
+
Koert Kuipers 2012-11-10, 19:42
+
Steve Loughran 2012-11-11, 18:04