Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka Hadoop Consumer issues


+
Abhi Basu 2013-11-07, 19:57
Copy link to this message
-
Re: Kafka Hadoop Consumer issues
Which class is not found?

Thanks,

Jun
On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <[EMAIL PROTECTED]> wrote:

> Let me describe my environment. Working on two nodes currently:
> 1.Single-node hadoop cluster (will refer as Node1)
> 2.Single node Kafka cluster  (will refer as Node2)
>
> Node 2 has 1 broker started with a topic (iot.test.stream) and one command
> line producer and one command line consumer to test the kafka install.
> Producer can send messages and the Consumer is receiving it.
>
> Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have edited
> the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file with the
> following:
>
> kafka.etl.topic=iot.test.stream
> hdfs.default.classpath.dir=/tmp/kafka/lib
> hadoop.job.ugi=kafka,hadoop
> kafka.server.uri=tcp://idh251-kafka:9095
> input=/tmp/kafka/data
> output=/tmp/kafka/output
> kafka.request.limit=-1
> ...........
>
> I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
>
> Next I run the following on Node 1:
> ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties from
> the /kafka-0.8/contrib/hadoop-consumer folder and get a
> classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
>
> What am I missing? I was thinking that running the sh file would allow me
> to retrieve messages with the same topic name to HDFS from Node 2 to Node
> 1. I just want to do an end to end test to see that messages coming into
> Kafka are being stored in HDFS with the minimal amount of code change
> required.
>
> Thanks,
>
> Abhi
>
> --
> Abhi Basu
>

 
+
Abhi Basu 2013-11-08, 13:24
+
Abhi Basu 2013-11-08, 16:40
+
Neha Narkhede 2013-11-08, 16:48
+
Abhi Basu 2013-11-08, 16:55
+
Abhi Basu 2013-11-08, 17:06
+
Abhi Basu 2013-11-08, 17:17
+
Abhi Basu 2013-11-08, 18:21
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB