Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> MiniMRCluster not behaving in hadoop 1.0.4


Copy link to this message
-
MiniMRCluster not behaving in hadoop 1.0.4
i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the
unit tests used to run fine but are now misbehaving. i do a simple setup in
the unit test like this:

    private static MiniDFSCluster dfsCluster;
    private static FileSystem fs;
    private static MiniMRCluster mrCluster;

    @BeforeClass
    public static void oneTimeSetUp() throws Exception {
        new File("tmp/logs").mkdirs();
        System.setProperty("hadoop.log.dir", "tmp/logs");
        System.setProperty("javax.xml.parsers.SAXParserFactory",

"com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl");
        final Configuration conf = new Configuration();
        conf.set("io.skip.checksum.errors", "true");
        dfsCluster = new MiniDFSCluster(conf, 2, true, null);
        fs = dfsCluster.getFileSystem();
        mrCluster = new MiniMRCluster(2,
dfsCluster.getFileSystem().getUri().toString(), 1, null, null, new
org.apache.hadoop.mapred.JobConf(conf));
    }

however with hadoop 1.0.4 this results in an exception where the jobtracker
dies:

12/11/10 02:38:29 ERROR mapred.MiniMRCluster: Job tracker crashed
java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
    at
org.apache.jasper.compiler.JspRuntimeContext.<init>(JspRuntimeContext.java:197)
    at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150)
    at
org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)

not sure what this error is about, but i managed to get around it for now
by removing the 2 hadoop apache jasper dependencies which seem to have
something to do with it: jasper-compiler-5.5.12.jar and
jasper-runtime-5.5.12.jar. with this "fix" i get further and my jobs
starts, but now all the tasks fail with messages like this:

attempt_20121110024037623_0001_r_000002_0: Exception in thread "main"
java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child

am i doing something completely wrong, and was i just lucky that this used
to run with CDH3? any ideas? thanks! koert
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB