Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> NullPointerException in Spring Data Hadoop with CDH4


Copy link to this message
-
NullPointerException in Spring Data Hadoop with CDH4
Hi,
I try to use Spring Data Hadoop with CDH4 to write a Map Reduce Job.

On startup, I get the following exception:

Exception in thread "SimpleAsyncTaskExecutor-1" java.lang.ExceptionInInitializerError
at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:183)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
at org.springframework.util.ReflectionUtils.makeAccessible(ReflectionUtils.java:405)
at org.springframework.data.hadoop.mapreduce.JobUtils.<clinit>(JobUtils.java:123)
... 2 more

I guess there is a problem with my Hadoop related dependencies. I couldn't find any reference showing how to configure Spring Data together with CDH4. But Costin showed, he is able to configure it: https://build.springsource.org/browse/SPRINGDATAHADOOP-CDH4-JOB1
**Maven Setup**

<properties>
<spring.hadoop.version>1.0.0.BUILD-SNAPSHOT</spring.hadoop.version>
<hadoop.version>2.0.0-cdh4.1.3</hadoop.version>
</properties>

<dependencies>
...
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-hadoop</artifactId>
<version>${spring.hadoop.version}</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-streaming</artifactId>
<version>${hadoop.version}</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-test</artifactId>
<version>2.0.0-mr1-cdh4.1.3</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-tools</artifactId>
<version>2.0.0-mr1-cdh4.1.3</version>
</dependency>
...
</dependencies>
...
<repositories>  
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>

<repository>
<id>spring-snapshot</id>
<name>Spring Maven SNAPSHOT Repository</name>
<url>http://repo.springframework.org/snapshot</url>
</repository>
</repositories>

**Application Context**

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:hdp="http://www.springframework.org/schema/hadoop" xmlns:context="http://www.springframework.org/schema/context"
xmlns:hadoop="http://www.springframework.org/schema/hadoop"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd
http://www.springframework.org/schema/context/spring-context.xsd http://www.springframework.org/schema/integration
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd">

<context:property-placeholder location="classpath:hadoop.properties" />

<hdp:configuration id="hadoopConfiguration">
fs.defaultFS=${hd.fs}
</hdp:configuration>

<hdp:job id="wordCountJob" input-path="${input.path}"
output-path="${output.path}" mapper="com.example.WordMapper"
reducer="com.example.WordReducer" />

<hdp:job-runner job-ref="wordCountJob" run-at-startup="true" wait-for-completion="true"/>

</beans>

**Cluster version**

Hadoop 2.0.0-cdh4.1.3
**Note:**

This small Unittest is running fine with the current configuration:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "classpath:/applicationContext.xml" })
public class Starter {

@Autowired
private Configuration configuration;

@Test
public void shellOps() {
Assert.assertNotNull(this.configuration);
FsShell fsShell = new FsShell(this.configuration);
final Collection<FileStatus> coll = fsShell.ls("/user");
System.out.println(coll);
}
}
It would be nice if someone can give me an example configuration.

Best Regards,
Christian.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB