Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Re: Thoughts on using Ant+Ivy for the build?


+
David Arthur 2013-04-08, 19:53
+
Jay Kreps 2013-04-08, 20:20
+
David Arthur 2013-04-08, 20:31
+
David Arthur 2013-04-08, 21:53
Copy link to this message
-
Thoughts on using Ant+Ivy for the build?
After getting frustrated with SBT, and being unable to figure out
seemingly simple problems like KAFKA-843, I decided to try something
completely different.

I spent some time yesterday adapting some Ant/Ivy boilerplate I use for
projects to Kafka. It was actually pretty easy to get working (Kafka is
a very simple build), and I think it's _much_ cleaner than the existing
SBT stuff.

Attached is a patchset of the work I did. N.B., this is totally
experimental and only considers the "core" part of the project.

At this point I can:

* Resolve all deps through Ivy (except yammer.metrics which is checked in)
* Resolve different versions of Scala through Ivy (i.e., cross
compile-ability)
* Compile the source
* Run all the unit tests (all passing)
* Compile a jar
* Package a tarball of the libs, conf, and bin scripts

Since all the libraries are localized to the project (and not in
~/.ivy2), the packaging is trivial. Also, the bin scripts could be
cleaned up significantly (which they need to be IMO).

I would love to hear what everyone thinks of this. Am I crazy? Is SBT
really better? Convince me!

-David
 kafka-ant-patchset.tar
0001-Compile-and-package-Kafka-with-Ant-Ivy.patch

From dcad0dea11aaa8aff407bccde468d2e7518e3b5b Mon Sep 17 00:00:00 2001
From: David Arthur <[EMAIL PROTECTED]>
Date: Thu, 4 Apr 2013 16:56:44 -0400
Subject: [PATCH 1/3] Compile and package Kafka with Ant+Ivy

 build.xml                                          |  130 ++++++++++++++++++++
 core/src/main/scala/kafka/utils/ZKConfig.scala     |   32 +++++
 .../scala/kafka/utils/ZKStringSerializer.scala     |   35 ++++++
 core/src/main/scala/kafka/utils/ZkUtils.scala      |   27 ----
 ivy.xml                                            |   38 ++++++
 ivy/ivysettings.xml                                |    6 +
 .../managed/metrics-annotation-3.0.0-c0c8be71.jar  |  Bin 0 -> 4766 bytes
 lib-ant/managed/metrics-core-3.0.0-c0c8be71.jar    |  Bin 0 -> 81782 bytes
 8 files changed, 241 insertions(+), 27 deletions(-)
 create mode 100644 build.xml
 create mode 100644 core/src/main/scala/kafka/utils/ZKConfig.scala
 create mode 100644 core/src/main/scala/kafka/utils/ZKStringSerializer.scala
 create mode 100644 ivy.xml
 create mode 100644 ivy/ivysettings.xml
 create mode 100644 lib-ant/managed/metrics-annotation-3.0.0-c0c8be71.jar
 create mode 100644 lib-ant/managed/metrics-core-3.0.0-c0c8be71.jar

diff --git a/build.xml b/build.xml
new file mode 100644
index 0000000..0f61d97
+++ b/build.xml
@@ -0,0 +1,130 @@
+<project name="kafka"
+         xmlns:artifact="antlib:org.apache.maven.artifact.ant"
+         xmlns:ivy="antlib:org.apache.ivy.ant">
+  <description>Apache Kafka</description>
+
+  <!-- Directories -->
+  <dirname property="base.dir" file="${ant.file.kafka}"/>
+
+  <!-- Properties -->
+  <property name="version" value="0.8.0"/>
+  <property name="debug" value="true"/>
+
+  <property name="ivy.dir" location="${base.dir}/ivy"/>
+  <property name="ivy.install.version" value="2.3.0-rc2"/>
+
+  <property name="build.dir" value="build-ant"/>
+  <property name="lib.dir" value="lib-ant"/>
+
+  <property name="src.dir" value="core/src/main/scala"/>
+  <property name="classes.dir" value="${build.dir}/classes"/>
+
+  <property name="assembly.dir" value="${build.dir}/assembly"/>
+
+  <!-- Ivy -->
+
+  <condition property="ivy.jar.exists">
+    <available file="${ivy.dir}/ivy.jar"/>
+  </condition>
+
+  <target name="ivy-download" unless="ivy.jar.exists">
+    <mkdir dir="${ivy.dir}"/>
+    <get src="http://repo2.maven.org/maven2/org/apache/ivy/ivy/${ivy.install.version}/ivy-${ivy.install.version}.jar"
+         dest="${ivy.dir}/ivy.jar"
+         usetimestamp="true"/>
+  </target>
+
+  <target name="ivy-init" depends="ivy-download" unless="skip.ivy" description="initialize ivy">
+    <path id="ivy.lib.path">
+      <fileset dir="${ivy.dir}" includes="*.jar"/>
+    </path>
+    <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpathref="ivy.lib.path"/>
+    <ivy:settings file="${ivy.dir}/ivysettings.xml"/>
+  </target>
+
+  <target name="ivy-clean-libs" depends="ivy-init" description="clean up ivy libs">
+    <delete includeemptydirs="true" quiet="true">
+      <fileset dir=".">
+        <include name="**/lib/ivy/**/*"/>
+      </fileset>
+    </delete>
+  </target>
+
+  <target name="ivy-clean" depends="ivy-init" description="clean ivy cache and libs">
+    <ivy:cleancache/>
+    <delete dir="${lib.dir}/ivy"/>
+  </target>
+
+  <target name="ivy-resolve" depends="ivy-init" description="fetch dependencies with ivy">
+    <ivy:resolve/>
+    <ivy:retrieve pattern="${lib.dir}/ivy/[conf]/[artifact]-[revision](-[classifier]).[ext]"/>
+    <ivy:report todir="${build.dir}/ivy-report"/>
+  </target>
+
+  <!-- Scala -->
+  <target name="scala-init" depends="ivy-init">
+    <ivy:resolve conf="scalac"/>
+    <ivy:retrieve conf="scalac" pattern="${lib.dir}/ivy/[conf]/[artifact]-[revision](-[classifier]).[ext]"/>
+    <path id="scala.lib.path">
+      <fileset dir="${lib.dir}/ivy/scalac" includes="*.jar"/>
+    </path>
+    <taskdef resource="scala/tools/ant/antlib.xml" classpathref="scala.lib.path"/>
+  </target>
+
+  <!-- Build -->
+
+  <target name="clean" description="clean up">
+    <delete dir="${build.dir}"/>
+    <delete dir="${lib.dir}/ivy"/>
+  </target>
+
+  <target name="compile" depends="ivy-resolve,scala-init" description="compile the source">
+    <mkdir dir="${classes.dir}"/>
+    <scalac srcdir="${src.dir}" destdir="${classes.dir}" encoding="UTF-8">
+      <classpath>
+        <fileset dir="${lib.dir}" includes="ivy/compile/*.jar"/>
+        <fileset dir="${lib.dir}" includes="managed/*.jar"/>
+        <fileset dir="${classes.dir}"/>
+      </classpath>
+      <include name="**/*.scala"/>
+      <include name="**/*.java"/>
+    </scalac>
+    <javac srcdir="${src.dir}" destdir="${classes.dir}" encoding="UTF-8">
+      <classpath>
+        <fileset dir="${lib.dir}" includes="ivy/compi
+
Jun Rao 2013-04-08, 05:06
+
David Arthur 2013-04-08, 12:44
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB