Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka Hadoop Consumer issues


Copy link to this message
-
Re: Kafka Hadoop Consumer issues
Neha Narkhede 2013-11-08, 16:48
ClassNotFound means the Hadoop job is not able to find the related jar.
Have you made sure the related jars are registered in the distributed cache?
On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <[EMAIL PROTECTED]> wrote:

> Can anyone help me with this issue? I feel like I am very close and am
> probably making some silly config error.
>
> Kafka team, please provide more detailed notes on how to make this
> component work.
>
> Thanks.
>
>
> On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <[EMAIL PROTECTED]> wrote:
>
> > Simplekafkaetljob class,  as mentioned in the post.
> >
> > Thanks
> >
> > Abhi
> >
> > From Samsung Galaxy S4
> > On Nov 7, 2013 8:34 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> >
> >> Which class is not found?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <[EMAIL PROTECTED]> wrote:
> >>
> >> > Let me describe my environment. Working on two nodes currently:
> >> > 1.Single-node hadoop cluster (will refer as Node1)
> >> > 2.Single node Kafka cluster  (will refer as Node2)
> >> >
> >> > Node 2 has 1 broker started with a topic (iot.test.stream) and one
> >> command
> >> > line producer and one command line consumer to test the kafka install.
> >> > Producer can send messages and the Consumer is receiving it.
> >> >
> >> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have
> >> edited
> >> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file with
> >> the
> >> > following:
> >> >
> >> > kafka.etl.topic=iot.test.stream
> >> > hdfs.default.classpath.dir=/tmp/kafka/lib
> >> > hadoop.job.ugi=kafka,hadoop
> >> > kafka.server.uri=tcp://idh251-kafka:9095
> >> > input=/tmp/kafka/data
> >> > output=/tmp/kafka/output
> >> > kafka.request.limit=-1
> >> > ...........
> >> >
> >> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
> >> >
> >> > Next I run the following on Node 1:
> >> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties
> >> from
> >> > the /kafka-0.8/contrib/hadoop-consumer folder and get a
> >> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
> >> >
> >> > What am I missing? I was thinking that running the sh file would allow
> >> me
> >> > to retrieve messages with the same topic name to HDFS from Node 2 to
> >> Node
> >> > 1. I just want to do an end to end test to see that messages coming
> into
> >> > Kafka are being stored in HDFS with the minimal amount of code change
> >> > required.
> >> >
> >> > Thanks,
> >> >
> >> > Abhi
> >> >
> >> > --
> >> > Abhi Basu
> >> >
> >>
> >
>
>
> --
> Abhi Basu
>