Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Thoughts on using Ant+Ivy for the build?


Copy link to this message
-
Thoughts on using Ant+Ivy for the build?
After getting frustrated with SBT, and being unable to figure out
seemingly simple problems like KAFKA-843, I decided to try something
completely different.

I spent some time yesterday adapting some Ant/Ivy boilerplate I use for
projects to Kafka. It was actually pretty easy to get working (Kafka is
a very simple build), and I think it's _much_ cleaner than the existing
SBT stuff.

Attached is a patchset of the work I did. N.B., this is totally
experimental and only considers the "core" part of the project.

At this point I can:

* Resolve all deps through Ivy (except yammer.metrics which is checked in)
* Resolve different versions of Scala through Ivy (i.e., cross
compile-ability)
* Compile the source
* Run all the unit tests (all passing)
* Compile a jar
* Package a tarball of the libs, conf, and bin scripts

Since all the libraries are localized to the project (and not in
~/.ivy2), the packaging is trivial. Also, the bin scripts could be
cleaned up significantly (which they need to be IMO).

I would love to hear what everyone thinks of this. Am I crazy? Is SBT
really better? Convince me!

-David
 kafka-ant-patchset.tar
0001-Compile-and-package-Kafka-with-Ant-Ivy.patch

From dcad0dea11aaa8aff407bccde468d2e7518e3b5b Mon Sep 17 00:00:00 2001
From: David Arthur <[EMAIL PROTECTED]>
Date: Thu, 4 Apr 2013 16:56:44 -0400
Subject: [PATCH 1/3] Compile and package Kafka with Ant+Ivy

---
 build.xml                                          |  130 ++++++++++++++++++++
 core/src/main/scala/kafka/utils/ZKConfig.scala     |   32 +++++
 .../scala/kafka/utils/ZKStringSerializer.scala     |   35 ++++++
 core/src/main/scala/kafka/utils/ZkUtils.scala      |   27 ----
 ivy.xml                                            |   38 ++++++
 ivy/ivysettings.xml                                |    6 +
 .../managed/metrics-annotation-3.0.0-c0c8be71.jar  |  Bin 0 -> 4766 bytes
 lib-ant/managed/metrics-core-3.0.0-c0c8be71.jar    |  Bin 0 -> 81782 bytes
 8 files changed, 241 insertions(+), 27 deletions(-)
 create mode 100644 build.xml
 create mode 100644 core/src/main/scala/kafka/utils/ZKConfig.scala
 create mode 100644 core/src/main/scala/kafka/utils/ZKStringSerializer.scala
 create mode 100644 ivy.xml
 create mode 100644 ivy/ivysettings.xml
 create mode 100644 lib-ant/managed/metrics-annotation-3.0.0-c0c8be71.jar
 create mode 100644 lib-ant/managed/metrics-core-3.0.0-c0c8be71.jar

diff --git a/build.xml b/build.xml
new file mode 100644
index 0000000..0f61d97
--- /dev/null
+++ b/build.xml
@@ -0,0 +1,130 @@
+<project name="kafka"
+         xmlns:artifact="antlib:org.apache.maven.artifact.ant"
+         xmlns:ivy="antlib:org.apache.ivy.ant">
+  <description>Apache Kafka</description>
+
+  <!-- Directories -->
+  <dirname property="base.dir" file="${ant.file.kafka}"/>
+
+  <!-- Properties -->
+  <property name="version" value="0.8.0"/>
+  <property name="debug" value="true"/>
+
+  <property name="ivy.dir" location="${base.dir}/ivy"/>
+  <property name="ivy.install.version" value="2.3.0-rc2"/>
+
+  <property name="build.dir" value="build-ant"/>
+  <property name="lib.dir" value="lib-ant"/>
+
+  <property name="src.dir" value="core/src/main/scala"/>
+  <property name="classes.dir" value="${build.dir}/classes"/>
+
+  <property name="assembly.dir" value="${build.dir}/assembly"/>
+
+  <!-- Ivy -->
+
+  <condition property="ivy.jar.exists">
+    <available file="${ivy.dir}/ivy.jar"/>
+  </condition>
+
+  <target name="ivy-download" unless="ivy.jar.exists">
+    <mkdir dir="${ivy.dir}"/>
+    <get src="http://repo2.maven.org/maven2/org/apache/ivy/ivy/${ivy.install.version}/ivy-${ivy.install.version}.jar"
+         dest="${ivy.dir}/ivy.jar"
+         usetimestamp="true"/>
+  </target>
+
+  <target name="ivy-init" depends="ivy-download" unless="skip.ivy" description="initialize ivy">
+    <path id="ivy.lib.path">
+      <fileset dir="${ivy.dir}" includes="*.jar"/>
+    </path>
+    <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpathref="ivy.lib.path"/>
+    <ivy:settings file="${ivy.dir}/ivysettings.xml"/>
+  </target>
+
+  <target name="ivy-clean-libs" depends="ivy-init" description="clean up ivy libs">
+    <delete includeemptydirs="true" quiet="true">
+      <fileset dir=".">
+        <include name="**/lib/ivy/**/*"/>
+      </fileset>
+    </delete>
+  </target>
+
+  <target name="ivy-clean" depends="ivy-init" description="clean ivy cache and libs">
+    <ivy:cleancache/>
+    <delete dir="${lib.dir}/ivy"/>
+  </target>
+
+  <target name="ivy-resolve" depends="ivy-init" description="fetch dependencies with ivy">
+    <ivy:resolve/>
+    <ivy:retrieve pattern="${lib.dir}/ivy/[conf]/[artifact]-[revision](-[classifier]).[ext]"/>
+    <ivy:report todir="${build.dir}/ivy-report"/>
+  </target>
+
+  <!-- Scala -->
+  <target name="scala-init" depends="ivy-init">
+    <ivy:resolve conf="scalac"/>
+    <ivy:retrieve conf="scalac" pattern="${lib.dir}/ivy/[conf]/[artifact]-[revision](-[classifier]).[ext]"/>
+    <path id="scala.lib.path">
+      <fileset dir="${lib.dir}/ivy/scalac" includes="*.jar"/>
+    </path>
+    <taskdef resource="scala/tools/ant/antlib.xml" classpathref="scala.lib.path"/>
+  </target>
+
+  <!-- Build -->
+
+  <target name="clean" description="clean up">
+    <delete dir="${build.dir}"/>
+    <delete dir="${lib.dir}/ivy"/>
+  </target>
+
+  <target name="compile" depends="ivy-resolve,scala-init" description="compile the source">
+    <mkdir dir="${classes.dir}"/>
+    <scalac srcdir="${src.dir}" destdir="${classes.dir}" encoding="UTF-8">
+      <classpath>
+        <fileset dir="${lib.dir}" includes="ivy/compile/*.jar"/>
+        <fileset dir="${lib.dir}" includes="managed/*.jar"/>
+        <fileset dir="${classes.dir}"/>
+      </classpath>
+      <include name="**/*.scala"/>
+      <include name="**/*.java"/>
+    </scalac>
+    <javac srcdir="${src.dir}" destdir="${classes.dir}" encoding="UTF-8">
+      <classpath>
+        <fileset dir="${lib.dir}" inc
+
Jun Rao 2013-04-08, 05:06
+
David Arthur 2013-04-08, 12:44
+
David Arthur 2013-04-08, 19:53
+
Jay Kreps 2013-04-08, 20:20
+
David Arthur 2013-04-08, 20:31
+
David Arthur 2013-04-08, 21:53