Let me describe my environment. Working on two nodes currently: 1.Single-node hadoop cluster (will refer as Node1) 2.Single node Kafka cluster (will refer as Node2)
Node 2 has 1 broker started with a topic (iot.test.stream) and one command line producer and one command line consumer to test the kafka install. Producer can send messages and the Consumer is receiving it.
Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have edited the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file with the following:
I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
Next I run the following on Node 1: ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties from the /kafka-0.8/contrib/hadoop-consumer folder and get a classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
What am I missing? I was thinking that running the sh file would allow me to retrieve messages with the same topic name to HDFS from Node 2 to Node 1. I just want to do an end to end test to see that messages coming into Kafka are being stored in HDFS with the minimal amount of code change required.
ClassNotFound means the Hadoop job is not able to find the related jar. Have you made sure the related jars are registered in the distributed cache? On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <[EMAIL PROTECTED]> wrote:
Still get the same error: [root@idh251-0 hadoop-consumer]# ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties Exception in thread "main" java.lang.NoClassDefFoundError: kafka/etl/impl/SimpleKafkaETLJob Caused by: java.lang.ClassNotFoundException: kafka.etl.impl.SimpleKafkaETLJob at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the main class: kafka.etl.impl.SimpleKafkaETLJob. Program will exit. [root@idh251-0 hadoop-consumer]# hadoop fs -ls /tmp/kafka/lib Warning: $HADOOP_HOME is deprecated.
Here is what is in the HDFS /tmp/kafka/lib folder after running the copy-jars.sh: