Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> MiniMRCluster not behaving in hadoop 1.0.4


Copy link to this message
-
Re: MiniMRCluster not behaving in hadoop 1.0.4
ant sets classpath for me but i have same issues when i set it myself.
for example i can run it like this:
java -cp lib/test/*:build/classes:build/test
com.tresata.hadoop.mapred.MapRedTest

my dependencies are hadoop-core, hadoop-test and i also had to add
jsr311api to make it work.
ls lib/test gives:

ant-1.6.5.jar
commons-beanutils-1.7.0.jar
commons-beanutils-core-1.8.0.jar
commons-cli-1.2.jar
commons-codec-1.4.jar
commons-collections-3.2.1.jar
commons-configuration-1.6.jar
commons-digester-1.8.jar
commons-el-1.0.jar
commons-httpclient-3.0.1.jar
commons-lang-2.4.jar
commons-logging-1.1.1.jar
commons-math-2.1.jar
commons-net-1.4.1.jar
core-3.1.1.jar
ftplet-api-1.0.0.jar
ftpserver-core-1.0.0.jar
ftpserver-deprecated-1.0.0-M2.jar
guava-r06.jar
hadoop-core-1.0.4.jar
hadoop-test-1.0.4.jar
hsqldb-1.8.0.10.jar
jackson-core-asl-1.0.1.jar
jackson-mapper-asl-1.0.1.jar
jasper-compiler-5.5.12.jar
jasper-runtime-5.5.12.jar
jets3t-0.7.1.jar
jetty-6.1.26.jar
jetty-util-6.1.26.jar
joda-time-2.1.jar
jsp-2.1-6.1.14.jar
jsp-api-2.1-6.1.14.jar
jsr311-api-1.1.1.jar
junit-3.8.1.jar
kfs-0.3.jar
log4j-1.2.16.jar
mina-core-2.0.0-M5.jar
oro-2.0.8.jar
servlet-api-2.5-20081211.jar
servlet-api-2.5-6.1.14.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
xmlenc-0.52.jar
On Sat, Nov 10, 2012 at 5:24 AM, Steve Loughran <[EMAIL PROTECTED]>wrote:

>
>
> On 10 November 2012 07:43, Koert Kuipers <[EMAIL PROTECTED]> wrote:
>
>> i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the
>> unit tests used to run fine but are now misbehaving. i do a simple setup in
>> the unit test like this:
>>
>>     private static MiniDFSCluster dfsCluster;
>>     private static FileSystem fs;
>>     private static MiniMRCluster mrCluster;
>>
>>     @BeforeClass
>>     public static void oneTimeSetUp() throws Exception {
>>         new File("tmp/logs").mkdirs();
>>         System.setProperty("hadoop.log.dir", "tmp/logs");
>>         System.setProperty("javax.xml.parsers.SAXParserFactory",
>>
>> "com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl");
>>         final Configuration conf = new Configuration();
>>         conf.set("io.skip.checksum.errors", "true");
>>         dfsCluster = new MiniDFSCluster(conf, 2, true, null);
>>         fs = dfsCluster.getFileSystem();
>>         mrCluster = new MiniMRCluster(2,
>> dfsCluster.getFileSystem().getUri().toString(), 1, null, null, new
>> org.apache.hadoop.mapred.JobConf(conf));
>>     }
>>
>> however with hadoop 1.0.4 this results in an exception where the
>> jobtracker dies:
>>
>> 12/11/10 02:38:29 ERROR mapred.MiniMRCluster: Job tracker crashed
>> java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
>>     at
>> org.apache.jasper.compiler.JspRuntimeContext.<init>(JspRuntimeContext.java:197)
>>     at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150)
>>     at
>> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
>>
>> not sure what this error is about, but i managed to get around it for now
>> by removing the 2 hadoop apache jasper dependencies which seem to have
>> something to do with it: jasper-compiler-5.5.12.jar and
>> jasper-runtime-5.5.12.jar. with this "fix" i get further and my jobs
>> starts, but now all the tasks fail with messages like this:
>>
>> attempt_20121110024037623_0001_r_000002_0: Exception in thread "main"
>> java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child
>>
>> am i doing something completely wrong, and was i just lucky that this
>> used to run with CDH3? any ideas? thanks! koert
>>
>>
> This sounds like some classpath/versioning thing -MiniMR cluster works for
> me; I even fixed that system property setup in trunk as I got fed up with
> having to do it myself.
>
> What is your CP in your test run?
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB