Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka Hadoop Consumer issues


+
Abhi Basu 2013-11-07, 19:57
Copy link to this message
-
Re: Kafka Hadoop Consumer issues
Which class is not found?

Thanks,

Jun
On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <[EMAIL PROTECTED]> wrote:

> Let me describe my environment. Working on two nodes currently:
> 1.Single-node hadoop cluster (will refer as Node1)
> 2.Single node Kafka cluster  (will refer as Node2)
>
> Node 2 has 1 broker started with a topic (iot.test.stream) and one command
> line producer and one command line consumer to test the kafka install.
> Producer can send messages and the Consumer is receiving it.
>
> Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have edited
> the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file with the
> following:
>
> kafka.etl.topic=iot.test.stream
> hdfs.default.classpath.dir=/tmp/kafka/lib
> hadoop.job.ugi=kafka,hadoop
> kafka.server.uri=tcp://idh251-kafka:9095
> input=/tmp/kafka/data
> output=/tmp/kafka/output
> kafka.request.limit=-1
> ...........
>
> I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
>
> Next I run the following on Node 1:
> ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties from
> the /kafka-0.8/contrib/hadoop-consumer folder and get a
> classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
>
> What am I missing? I was thinking that running the sh file would allow me
> to retrieve messages with the same topic name to HDFS from Node 2 to Node
> 1. I just want to do an end to end test to see that messages coming into
> Kafka are being stored in HDFS with the minimal amount of code change
> required.
>
> Thanks,
>
> Abhi
>
> --
> Abhi Basu
>

 
+
Abhi Basu 2013-11-08, 13:24
+
Abhi Basu 2013-11-08, 16:40
+
Neha Narkhede 2013-11-08, 16:48
+
Abhi Basu 2013-11-08, 16:55
+
Abhi Basu 2013-11-08, 17:06
+
Abhi Basu 2013-11-08, 17:17
+
Abhi Basu 2013-11-08, 18:21