Kafka, mail # user - Re: Is anyone able to consume from Kafka 0.7.x and write into Hadoop CDH 4.x ?

Is anyone able to consume from Kafka 0.7.x and write into Hadoop CDH 4.x ?

Felix GV 2013-01-07, 19:21

Hello all,

I haven't been reading the list for the past couple weeks, I've quite
busy... but I've searched and didn't find any discussions related to my
current issue, so I thought I'd ask while I'm still investigating on my
own...!

We've been running a Kafka 0.7.0 cluster without problem for a while now.
I've played around<http://felixgv.com/post/88/kafka-distributed-incremental-hadoop-consumer/>with
importing data from our Kafka cluster into hadoop a while ago, using
the simple Kafka consumer located in the contrib directory of the Kafka
source, and that worked properly. At the time, the Hadoop cluster I was
running was CDH3u3, IIRC.

I'm now revisiting that project with a brand new CDH4.1.2 Hadoop cluster
(using MR1, not YARN), and I'm having difficulty getting it to work.

At first, the run-class.sh script in kafka/contrib/hadoop-consumer wasn't
using the proper hadoop jars to connect to my cluster, so I tweaked it so
that it includes the output of the `hadoop classpath` command in its
classpath. It's now able to connect to my hadoop cluster, but it's telling
me that the versions don't match:

Exception in thread "main" org.apache.hadoop.ipc.RemoteException: Server
IPC version 7 cannot communicate with client version 3
at org.apache.hadoop.ipc.Client.call(Client.java:740)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getProtocolVersion(Unknown Source)
... (I could give the whole stacktrace if you want, but I didn't think
that's really relevant...)

So anyway, I've messed around with the
kafka/project/build/KafkaProject.scala file so that it uses the
"2.0.0-mr1-cdh4.1.2" version of hadoop-core, and fetches it from the
cloudera repo. I've added the cloudera repo by adding this line at the
beginning of the HadoopConsumerProject class section:

val clouderaRepo = "Cloudera" at "
https://repository.cloudera.com/artifactory/cloudera-repos/"

When I run ./sbt update, it fetches the new jars correctly, but then, when
I run ./sbt package, it's not able to find a bunch of hadoop related
classes and packages in the hadoop-consumer code, which I guess means that
a few APIs have changed between the two versions of CDH.

I've tried this on the 0.7.0 branch of Kafka (from the Apache git repo) as
well as on the 0.7.2 branch, and I get the same result on both (I can't
successfully run ./sbt package). The easiest for me would be to get it to
work on Kafka 0.7.0, but I guess I could persuade my people to upgrade to
0.7.2 if it's necessary (I'd like us to upgrade, but I guess you all know
how it is... getting a working system to change is a political hassle). I
don't think we'd be willing to move to Kafka 0.8 just yet, so hopefully
that won't be necessary.

*TLDR: Is anyone pumping data from Kafka 0.7.x to CDH4.x ? And if so, how?
Using the example consumer from kafka's contrib, or another one?* Perhaps this
one <https://github.com/miniway/kafka-hadoop-consumer>? (I'll probably give
it a try soon, BTW, so I'll keep you guys posted...). I may also try
porting the hadoop-consumer contrib to CDH4.

Finally, I haven't seen anything mentioned about the LinkedIn
kafka/avro/hadoop ETL stuff we've been hearing about for a while. I saw the
new LinkedIn DataFu stuff but it seems unrelated. Are there any updates
about whether or when the ETL code would get open sourced? As far as we're
concerned, we're using avro quite a bit, so in our case, the avro coupling
would definitely not be an issue. I don't know what version(s) of hadoop
LinkedIn is running, though, so perhaps their stuff wouldn't work out of
the box with CDH4 either anyway...

Any advice would be appreciated!

Thanks :) !

--
Felix