Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka Hadoop Consumer issues


+
Abhi Basu 2013-11-07, 19:57
+
Jun Rao 2013-11-08, 04:34
Copy link to this message
-
Re: Kafka Hadoop Consumer issues
Simplekafkaetljob class,  as mentioned in the post.

Thanks

Abhi

From Samsung Galaxy S4
On Nov 7, 2013 8:34 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:

> Which class is not found?
>
> Thanks,
>
> Jun
>
>
> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <[EMAIL PROTECTED]> wrote:
>
> > Let me describe my environment. Working on two nodes currently:
> > 1.Single-node hadoop cluster (will refer as Node1)
> > 2.Single node Kafka cluster  (will refer as Node2)
> >
> > Node 2 has 1 broker started with a topic (iot.test.stream) and one
> command
> > line producer and one command line consumer to test the kafka install.
> > Producer can send messages and the Consumer is receiving it.
> >
> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have edited
> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file with the
> > following:
> >
> > kafka.etl.topic=iot.test.stream
> > hdfs.default.classpath.dir=/tmp/kafka/lib
> > hadoop.job.ugi=kafka,hadoop
> > kafka.server.uri=tcp://idh251-kafka:9095
> > input=/tmp/kafka/data
> > output=/tmp/kafka/output
> > kafka.request.limit=-1
> > ...........
> >
> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
> >
> > Next I run the following on Node 1:
> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties from
> > the /kafka-0.8/contrib/hadoop-consumer folder and get a
> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
> >
> > What am I missing? I was thinking that running the sh file would allow me
> > to retrieve messages with the same topic name to HDFS from Node 2 to Node
> > 1. I just want to do an end to end test to see that messages coming into
> > Kafka are being stored in HDFS with the minimal amount of code change
> > required.
> >
> > Thanks,
> >
> > Abhi
> >
> > --
> > Abhi Basu
> >
>

 
+
Abhi Basu 2013-11-08, 16:40
+
Neha Narkhede 2013-11-08, 16:48
+
Abhi Basu 2013-11-08, 16:55
+
Abhi Basu 2013-11-08, 17:06
+
Abhi Basu 2013-11-08, 17:17
+
Abhi Basu 2013-11-08, 18:21