Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> MiniMRCluster not behaving in hadoop 1.0.4


Copy link to this message
-
Re: MiniMRCluster not behaving in hadoop 1.0.4
I don't see anything that obvious, except there are two servlet API JARS
-one of those should go

On 10 November 2012 19:42, Koert Kuipers <[EMAIL PROTECTED]> wrote:

> ant sets classpath for me but i have same issues when i set it myself.
> for example i can run it like this:
> java -cp lib/test/*:build/classes:build/test
> com.tresata.hadoop.mapred.MapRedTest
>
> my dependencies are hadoop-core, hadoop-test and i also had to add
> jsr311api to make it work.
> ls lib/test gives:
>
> ant-1.6.5.jar
> commons-beanutils-1.7.0.jar
> commons-beanutils-core-1.8.0.jar
> commons-cli-1.2.jar
> commons-codec-1.4.jar
> commons-collections-3.2.1.jar
> commons-configuration-1.6.jar
> commons-digester-1.8.jar
> commons-el-1.0.jar
> commons-httpclient-3.0.1.jar
> commons-lang-2.4.jar
> commons-logging-1.1.1.jar
> commons-math-2.1.jar
> commons-net-1.4.1.jar
> core-3.1.1.jar
> ftplet-api-1.0.0.jar
> ftpserver-core-1.0.0.jar
> ftpserver-deprecated-1.0.0-M2.jar
> guava-r06.jar
> hadoop-core-1.0.4.jar
> hadoop-test-1.0.4.jar
> hsqldb-1.8.0.10.jar
> jackson-core-asl-1.0.1.jar
> jackson-mapper-asl-1.0.1.jar
> jasper-compiler-5.5.12.jar
> jasper-runtime-5.5.12.jar
> jets3t-0.7.1.jar
> jetty-6.1.26.jar
> jetty-util-6.1.26.jar
> joda-time-2.1.jar
> jsp-2.1-6.1.14.jar
> jsp-api-2.1-6.1.14.jar
> jsr311-api-1.1.1.jar
> junit-3.8.1.jar
> kfs-0.3.jar
> log4j-1.2.16.jar
> mina-core-2.0.0-M5.jar
> oro-2.0.8.jar
> servlet-api-2.5-20081211.jar
> servlet-api-2.5-6.1.14.jar
> slf4j-api-1.6.1.jar
> slf4j-log4j12-1.6.1.jar
> xmlenc-0.52.jar
>
>
> On Sat, Nov 10, 2012 at 5:24 AM, Steve Loughran <[EMAIL PROTECTED]>wrote:
>
>>
>>
>> On 10 November 2012 07:43, Koert Kuipers <[EMAIL PROTECTED]> wrote:
>>
>>> i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the
>>> unit tests used to run fine but are now misbehaving. i do a simple setup in
>>> the unit test like this:
>>>
>>>     private static MiniDFSCluster dfsCluster;
>>>     private static FileSystem fs;
>>>     private static MiniMRCluster mrCluster;
>>>
>>>     @BeforeClass
>>>     public static void oneTimeSetUp() throws Exception {
>>>         new File("tmp/logs").mkdirs();
>>>         System.setProperty("hadoop.log.dir", "tmp/logs");
>>>         System.setProperty("javax.xml.parsers.SAXParserFactory",
>>>
>>> "com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl");
>>>         final Configuration conf = new Configuration();
>>>         conf.set("io.skip.checksum.errors", "true");
>>>         dfsCluster = new MiniDFSCluster(conf, 2, true, null);
>>>         fs = dfsCluster.getFileSystem();
>>>         mrCluster = new MiniMRCluster(2,
>>> dfsCluster.getFileSystem().getUri().toString(), 1, null, null, new
>>> org.apache.hadoop.mapred.JobConf(conf));
>>>     }
>>>
>>> however with hadoop 1.0.4 this results in an exception where the
>>> jobtracker dies:
>>>
>>> 12/11/10 02:38:29 ERROR mapred.MiniMRCluster: Job tracker crashed
>>> java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
>>>     at
>>> org.apache.jasper.compiler.JspRuntimeContext.<init>(JspRuntimeContext.java:197)
>>>     at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150)
>>>     at
>>> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
>>>
>>> not sure what this error is about, but i managed to get around it for
>>> now by removing the 2 hadoop apache jasper dependencies which seem to have
>>> something to do with it: jasper-compiler-5.5.12.jar and
>>> jasper-runtime-5.5.12.jar. with this "fix" i get further and my jobs
>>> starts, but now all the tasks fail with messages like this:
>>>
>>> attempt_20121110024037623_0001_r_000002_0: Exception in thread "main"
>>> java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child
>>>
>>> am i doing something completely wrong, and was i just lucky that this
>>> used to run with CDH3? any ideas? thanks! koert
>>>
>>>
>> This sounds like some classpath/versioning thing -MiniMR cluster works
>> for me; I even fixed that system property setup in trunk as I got fed up
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB