Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> MiniMRCluster not behaving in hadoop 1.0.4


Copy link to this message
-
Re: MiniMRCluster not behaving in hadoop 1.0.4
On 10 November 2012 07:43, Koert Kuipers <[EMAIL PROTECTED]> wrote:

> i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the
> unit tests used to run fine but are now misbehaving. i do a simple setup in
> the unit test like this:
>
>     private static MiniDFSCluster dfsCluster;
>     private static FileSystem fs;
>     private static MiniMRCluster mrCluster;
>
>     @BeforeClass
>     public static void oneTimeSetUp() throws Exception {
>         new File("tmp/logs").mkdirs();
>         System.setProperty("hadoop.log.dir", "tmp/logs");
>         System.setProperty("javax.xml.parsers.SAXParserFactory",
>
> "com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl");
>         final Configuration conf = new Configuration();
>         conf.set("io.skip.checksum.errors", "true");
>         dfsCluster = new MiniDFSCluster(conf, 2, true, null);
>         fs = dfsCluster.getFileSystem();
>         mrCluster = new MiniMRCluster(2,
> dfsCluster.getFileSystem().getUri().toString(), 1, null, null, new
> org.apache.hadoop.mapred.JobConf(conf));
>     }
>
> however with hadoop 1.0.4 this results in an exception where the
> jobtracker dies:
>
> 12/11/10 02:38:29 ERROR mapred.MiniMRCluster: Job tracker crashed
> java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
>     at
> org.apache.jasper.compiler.JspRuntimeContext.<init>(JspRuntimeContext.java:197)
>     at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150)
>     at
> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
>
> not sure what this error is about, but i managed to get around it for now
> by removing the 2 hadoop apache jasper dependencies which seem to have
> something to do with it: jasper-compiler-5.5.12.jar and
> jasper-runtime-5.5.12.jar. with this "fix" i get further and my jobs
> starts, but now all the tasks fail with messages like this:
>
> attempt_20121110024037623_0001_r_000002_0: Exception in thread "main"
> java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child
>
> am i doing something completely wrong, and was i just lucky that this used
> to run with CDH3? any ideas? thanks! koert
>
>
This sounds like some classpath/versioning thing -MiniMR cluster works for
me; I even fixed that system property setup in trunk as I got fed up with
having to do it myself.

What is your CP in your test run?