Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> MiniMRCluster not behaving in hadoop 1.0.4


Copy link to this message
-
Re: MiniMRCluster not behaving in hadoop 1.0.4
On 10 November 2012 07:43, Koert Kuipers <[EMAIL PROTECTED]> wrote:

> i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the
> unit tests used to run fine but are now misbehaving. i do a simple setup in
> the unit test like this:
>
>     private static MiniDFSCluster dfsCluster;
>     private static FileSystem fs;
>     private static MiniMRCluster mrCluster;
>
>     @BeforeClass
>     public static void oneTimeSetUp() throws Exception {
>         new File("tmp/logs").mkdirs();
>         System.setProperty("hadoop.log.dir", "tmp/logs");
>         System.setProperty("javax.xml.parsers.SAXParserFactory",
>
> "com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl");
>         final Configuration conf = new Configuration();
>         conf.set("io.skip.checksum.errors", "true");
>         dfsCluster = new MiniDFSCluster(conf, 2, true, null);
>         fs = dfsCluster.getFileSystem();
>         mrCluster = new MiniMRCluster(2,
> dfsCluster.getFileSystem().getUri().toString(), 1, null, null, new
> org.apache.hadoop.mapred.JobConf(conf));
>     }
>
> however with hadoop 1.0.4 this results in an exception where the
> jobtracker dies:
>
> 12/11/10 02:38:29 ERROR mapred.MiniMRCluster: Job tracker crashed
> java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
>     at
> org.apache.jasper.compiler.JspRuntimeContext.<init>(JspRuntimeContext.java:197)
>     at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150)
>     at
> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
>
> not sure what this error is about, but i managed to get around it for now
> by removing the 2 hadoop apache jasper dependencies which seem to have
> something to do with it: jasper-compiler-5.5.12.jar and
> jasper-runtime-5.5.12.jar. with this "fix" i get further and my jobs
> starts, but now all the tasks fail with messages like this:
>
> attempt_20121110024037623_0001_r_000002_0: Exception in thread "main"
> java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child
>
> am i doing something completely wrong, and was i just lucky that this used
> to run with CDH3? any ideas? thanks! koert
>
>
This sounds like some classpath/versioning thing -MiniMR cluster works for
me; I even fixed that system property setup in trunk as I got fed up with
having to do it myself.

What is your CP in your test run?
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB