Hi,
  Im writing a framework involving kafka consumers in  scala and Im using sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it available in any  maven or ivey repositories?
Sam William
[EMAIL PROTECTED]

run the command below
./sbt

update
package

and then you will get a kafka.jar

On Fri, May 25, 2012 at 1:34 PM, Sam William <[EMAIL PROTECTED]> wrote:

> Hi,
>  Im writing a framework involving kafka consumers in  scala and Im using
> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
> available in any  maven or ivey repositories?
>
>
> Sam William
> [EMAIL PROTECTED]
>
>
>
>

Sam, I couldn't find it in a public repository, but maybe I didn't look
hard enough. I ended up deploying it into our own private repo. Good luck.

On Fri, May 25, 2012 at 1:30 AM, Bo Sun <[EMAIL PROTECTED]> wrote:

> run the command below
> ./sbt
>
> update
> package
>
> and then you will get a kafka.jar
>
> On Fri, May 25, 2012 at 1:34 PM, Sam William <[EMAIL PROTECTED]>
> wrote:
>
> > Hi,
> >  Im writing a framework involving kafka consumers in  scala and Im using
> > sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
> > available in any  maven or ivey repositories?
> >
> >
> > Sam William
> > [EMAIL PROTECTED]
> >
> >
> >
> >
>

Sam,

Because of the overhead of certifying every dependent jar, we chose to do
only the source release, not the binary release for Kafka. This creates a
bit overhead for our users since they have to build the source to generate
the kafka jar.

Thanks,

Jun

On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]> wrote:

> Hi,
>  Im writing a framework involving kafka consumers in  scala and Im using
> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
> available in any  maven or ivey repositories?
>
>
> Sam William
> [EMAIL PROTECTED]
>
>
>
>

Hi Jun,

Here's my +1 for publishing Kafka (jar and pom.xml with dependencies) to some public repo.

That makes it _much_ easier for a project to include Kafka.

Note that it's pretty easy to get it into the Apache repo; publishing to Maven central is a bit more challenging...

-- Ken

On May 25, 2012, at 9:46am, Jun Rao wrote:

> Sam,
>
> Because of the overhead of certifying every dependent jar, we chose to do
> only the source release, not the binary release for Kafka. This creates a
> bit overhead for our users since they have to build the source to generate
> the kafka jar.
>
> Thanks,
>
> Jun
>
> On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>> Im writing a framework involving kafka consumers in  scala and Im using
>> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
>> available in any  maven or ivey repositories?
>>
>>
>> Sam William
>> [EMAIL PROTECTED]
>>
>>
>>
>>

--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr

Ken,

We tried to do a binary release and ended up going through 10 RCs, mostly
to sort out the license of all jars. We gave up in the end. One of the
biggest problems is that we have contribs for clients in other languages,
which introduces dependent libraries in C#, go, etc whose license agreement
is not totally clear to us. If someone can help out on that, we'd be happy
to do a binary release.

Thanks,

Jun

On Fri, May 25, 2012 at 10:16 AM, Ken Krugler
<[EMAIL PROTECTED]>wrote:

> Hi Jun,
>
> Here's my +1 for publishing Kafka (jar and pom.xml with dependencies) to
> some public repo.
>
> That makes it _much_ easier for a project to include Kafka.
>
> Note that it's pretty easy to get it into the Apache repo; publishing to
> Maven central is a bit more challenging...
>
> -- Ken
>
> On May 25, 2012, at 9:46am, Jun Rao wrote:
>
> > Sam,
> >
> > Because of the overhead of certifying every dependent jar, we chose to do
> > only the source release, not the binary release for Kafka. This creates a
> > bit overhead for our users since they have to build the source to
> generate
> > the kafka jar.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi,
> >> Im writing a framework involving kafka consumers in  scala and Im using
> >> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
> >> available in any  maven or ivey repositories?
> >>
> >>
> >> Sam William
> >> [EMAIL PROTECTED]
> >>
> >>
> >>
> >>
>
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Mahout & Solr
>
>
>
>
>

Hi Jun,

On May 25, 2012, at 10:34am, Jun Rao wrote:

> Ken,
>
> We tried to do a binary release and ended up going through 10 RCs, mostly
> to sort out the license of all jars. We gave up in the end. One of the
> biggest problems is that we have contribs for clients in other languages,
> which introduces dependent libraries in C#, go, etc whose license agreement
> is not totally clear to us. If someone can help out on that, we'd be happy
> to do a binary release.

If you send me the list of dependencies that the Kafka jar has, I could help out.

From the test project I created, it looks like kafka-0.7.0.jar depends on:

- scala-library.jar
- zkclient.jar

So for Java users (who'd care about being able to pull kafka from a repo) are there any other dependencies?

Thanks,

-- Ken

> On Fri, May 25, 2012 at 10:16 AM, Ken Krugler
> <[EMAIL PROTECTED]>wrote:
>
>> Hi Jun,
>>
>> Here's my +1 for publishing Kafka (jar and pom.xml with dependencies) to
>> some public repo.
>>
>> That makes it _much_ easier for a project to include Kafka.
>>
>> Note that it's pretty easy to get it into the Apache repo; publishing to
>> Maven central is a bit more challenging...
>>
>> -- Ken
>>
>> On May 25, 2012, at 9:46am, Jun Rao wrote:
>>
>>> Sam,
>>>
>>> Because of the overhead of certifying every dependent jar, we chose to do
>>> only the source release, not the binary release for Kafka. This creates a
>>> bit overhead for our users since they have to build the source to
>> generate
>>> the kafka jar.
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]>
>> wrote:
>>>
>>>> Hi,
>>>> Im writing a framework involving kafka consumers in  scala and Im using
>>>> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
>>>> available in any  maven or ivey repositories?
>>>>
>>>>
>>>> Sam William
>>>> [EMAIL PROTECTED]
>>>>
>>>>
>>>>
>>>>
>>
>> --------------------------
>> Ken Krugler
>> http://www.scaleunlimited.com
>> custom big data solutions & training
>> Hadoop, Cascading, Mahout & Solr
>>
>>
>>
>>
>>

--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr

Im a bit reluctant to set up a new maven repo server right now . However , I expected sbt to pick  the jars that are available in my local .m2 repo .  I have kafka.07.jar installed on my local .m2 repo but sbt still cant find it .  Sorry Im new to sbt.  Whats the best way to make sbt pick up local jars ?

On May 25, 2012, at 11:13 AM, Ken Krugler wrote:

> Hi Jun,
>
> On May 25, 2012, at 10:34am, Jun Rao wrote:
>
>> Ken,
>>
>> We tried to do a binary release and ended up going through 10 RCs, mostly
>> to sort out the license of all jars. We gave up in the end. One of the
>> biggest problems is that we have contribs for clients in other languages,
>> which introduces dependent libraries in C#, go, etc whose license agreement
>> is not totally clear to us. If someone can help out on that, we'd be happy
>> to do a binary release.
>
> If you send me the list of dependencies that the Kafka jar has, I could help out.
>
> From the test project I created, it looks like kafka-0.7.0.jar depends on:
>
> - scala-library.jar
> - zkclient.jar
>
> So for Java users (who'd care about being able to pull kafka from a repo) are there any other dependencies?
>
> Thanks,
>
> -- Ken
>
>
>
>> On Fri, May 25, 2012 at 10:16 AM, Ken Krugler
>> <[EMAIL PROTECTED]>wrote:
>>
>>> Hi Jun,
>>>
>>> Here's my +1 for publishing Kafka (jar and pom.xml with dependencies) to
>>> some public repo.
>>>
>>> That makes it _much_ easier for a project to include Kafka.
>>>
>>> Note that it's pretty easy to get it into the Apache repo; publishing to
>>> Maven central is a bit more challenging...
>>>
>>> -- Ken
>>>
>>> On May 25, 2012, at 9:46am, Jun Rao wrote:
>>>
>>>> Sam,
>>>>
>>>> Because of the overhead of certifying every dependent jar, we chose to do
>>>> only the source release, not the binary release for Kafka. This creates a
>>>> bit overhead for our users since they have to build the source to
>>> generate
>>>> the kafka jar.
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>> On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]>
>>> wrote:
>>>>
>>>>> Hi,
>>>>> Im writing a framework involving kafka consumers in  scala and Im using
>>>>> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
>>>>> available in any  maven or ivey repositories?
>>>>>
>>>>>
>>>>> Sam William
>>>>> [EMAIL PROTECTED]
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>> --------------------------
>>> Ken Krugler
>>> http://www.scaleunlimited.com
>>> custom big data solutions & training
>>> Hadoop, Cascading, Mahout & Solr
>>>
>>>
>>>
>>>
>>>
>
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Mahout & Solr
>
>
>
>

Sam William
[EMAIL PROTECTED]

Sbt uses ivy, it usually looks in your .ivy2/cache   .

-Evan
Carry your candle, run to the darkness
Seek out the helpless, deceived and poor
Hold out your candle for all to see it
Take your candle, and go light your world
 
On May 25, 2012, at 11:28 AM, Sam William <[EMAIL PROTECTED]> wrote:

> Im a bit reluctant to set up a new maven repo server right now . However , I expected sbt to pick  the jars that are available in my local .m2 repo .  I have kafka.07.jar installed on my local .m2 repo but sbt still cant find it .  Sorry Im new to sbt.  Whats the best way to make sbt pick up local jars ?
>
> On May 25, 2012, at 11:13 AM, Ken Krugler wrote:
>
>> Hi Jun,
>>
>> On May 25, 2012, at 10:34am, Jun Rao wrote:
>>
>>> Ken,
>>>
>>> We tried to do a binary release and ended up going through 10 RCs, mostly
>>> to sort out the license of all jars. We gave up in the end. One of the
>>> biggest problems is that we have contribs for clients in other languages,
>>> which introduces dependent libraries in C#, go, etc whose license agreement
>>> is not totally clear to us. If someone can help out on that, we'd be happy
>>> to do a binary release.
>>
>> If you send me the list of dependencies that the Kafka jar has, I could help out.
>>
>> From the test project I created, it looks like kafka-0.7.0.jar depends on:
>>
>> - scala-library.jar
>> - zkclient.jar
>>
>> So for Java users (who'd care about being able to pull kafka from a repo) are there any other dependencies?
>>
>> Thanks,
>>
>> -- Ken
>>
>>
>>
>>> On Fri, May 25, 2012 at 10:16 AM, Ken Krugler
>>> <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hi Jun,
>>>>
>>>> Here's my +1 for publishing Kafka (jar and pom.xml with dependencies) to
>>>> some public repo.
>>>>
>>>> That makes it _much_ easier for a project to include Kafka.
>>>>
>>>> Note that it's pretty easy to get it into the Apache repo; publishing to
>>>> Maven central is a bit more challenging...
>>>>
>>>> -- Ken
>>>>
>>>> On May 25, 2012, at 9:46am, Jun Rao wrote:
>>>>
>>>>> Sam,
>>>>>
>>>>> Because of the overhead of certifying every dependent jar, we chose to do
>>>>> only the source release, not the binary release for Kafka. This creates a
>>>>> bit overhead for our users since they have to build the source to
>>>> generate
>>>>> the kafka jar.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]>
>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> Im writing a framework involving kafka consumers in  scala and Im using
>>>>>> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
>>>>>> available in any  maven or ivey repositories?
>>>>>>
>>>>>>
>>>>>> Sam William
>>>>>> [EMAIL PROTECTED]
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --------------------------
>>>> Ken Krugler
>>>> http://www.scaleunlimited.com
>>>> custom big data solutions & training
>>>> Hadoop, Cascading, Mahout & Solr
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>> --------------------------
>> Ken Krugler
>> http://www.scaleunlimited.com
>> custom big data solutions & training
>> Hadoop, Cascading, Mahout & Solr
>>
>>
>>
>>
>
> Sam William
> [EMAIL PROTECTED]
>
>
>

Ken,

If you do ./sbt update and ./sbt package, you will find all the jars and
ddls needed. The core Kafka project only depends on zookeeper, zkclient and
scala. However, other contribs depend on a lot of more stuff.

Thanks,

Jun

On Fri, May 25, 2012 at 11:13 AM, Ken Krugler
<[EMAIL PROTECTED]>wrote:

> Hi Jun,
>
> On May 25, 2012, at 10:34am, Jun Rao wrote:
>
> > Ken,
> >
> > We tried to do a binary release and ended up going through 10 RCs, mostly
> > to sort out the license of all jars. We gave up in the end. One of the
> > biggest problems is that we have contribs for clients in other languages,
> > which introduces dependent libraries in C#, go, etc whose license
> agreement
> > is not totally clear to us. If someone can help out on that, we'd be
> happy
> > to do a binary release.
>
> If you send me the list of dependencies that the Kafka jar has, I could
> help out.
>
> From the test project I created, it looks like kafka-0.7.0.jar depends on:
>
> - scala-library.jar
> - zkclient.jar
>
> So for Java users (who'd care about being able to pull kafka from a repo)
> are there any other dependencies?
>
> Thanks,
>
> -- Ken
>
>
>
> > On Fri, May 25, 2012 at 10:16 AM, Ken Krugler
> > <[EMAIL PROTECTED]>wrote:
> >
> >> Hi Jun,
> >>
> >> Here's my +1 for publishing Kafka (jar and pom.xml with dependencies) to
> >> some public repo.
> >>
> >> That makes it _much_ easier for a project to include Kafka.
> >>
> >> Note that it's pretty easy to get it into the Apache repo; publishing to
> >> Maven central is a bit more challenging...
> >>
> >> -- Ken
> >>
> >> On May 25, 2012, at 9:46am, Jun Rao wrote:
> >>
> >>> Sam,
> >>>
> >>> Because of the overhead of certifying every dependent jar, we chose to
> do
> >>> only the source release, not the binary release for Kafka. This
> creates a
> >>> bit overhead for our users since they have to build the source to
> >> generate
> >>> the kafka jar.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]>
> >> wrote:
> >>>
> >>>> Hi,
> >>>> Im writing a framework involving kafka consumers in  scala and Im
> using
> >>>> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
> >>>> available in any  maven or ivey repositories?
> >>>>
> >>>>
> >>>> Sam William
> >>>> [EMAIL PROTECTED]
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> >> --------------------------
> >> Ken Krugler
> >> http://www.scaleunlimited.com
> >> custom big data solutions & training
> >> Hadoop, Cascading, Mahout & Solr
> >>
> >>
> >>
> >>
> >>
>
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Mahout & Solr
>
>
>
>
>

Jun,

What if the contrib and other clients were split off into their own projects?  This would help achieve binary release for core Kafka, plus enable faster development of the other projects (for example, I could easily fork and improve on the ruby client without worrying about the other code...)

-Evan
Carry your candle, run to the darkness
Seek out the helpless, deceived and poor
Hold out your candle for all to see it
Take your candle, and go light your world
 
On May 25, 2012, at 10:34 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Ken,
>
> We tried to do a binary release and ended up going through 10 RCs, mostly
> to sort out the license of all jars. We gave up in the end. One of the
> biggest problems is that we have contribs for clients in other languages,
> which introduces dependent libraries in C#, go, etc whose license agreement
> is not totally clear to us. If someone can help out on that, we'd be happy
> to do a binary release.
>
> Thanks,
>
> Jun
>
> On Fri, May 25, 2012 at 10:16 AM, Ken Krugler
> <[EMAIL PROTECTED]>wrote:
>
>> Hi Jun,
>>
>> Here's my +1 for publishing Kafka (jar and pom.xml with dependencies) to
>> some public repo.
>>
>> That makes it _much_ easier for a project to include Kafka.
>>
>> Note that it's pretty easy to get it into the Apache repo; publishing to
>> Maven central is a bit more challenging...
>>
>> -- Ken
>>
>> On May 25, 2012, at 9:46am, Jun Rao wrote:
>>
>>> Sam,
>>>
>>> Because of the overhead of certifying every dependent jar, we chose to do
>>> only the source release, not the binary release for Kafka. This creates a
>>> bit overhead for our users since they have to build the source to
>> generate
>>> the kafka jar.
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]>
>> wrote:
>>>
>>>> Hi,
>>>> Im writing a framework involving kafka consumers in  scala and Im using
>>>> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
>>>> available in any  maven or ivey repositories?
>>>>
>>>>
>>>> Sam William
>>>> [EMAIL PROTECTED]
>>>>
>>>>
>>>>
>>>>
>>
>> --------------------------
>> Ken Krugler
>> http://www.scaleunlimited.com
>> custom big data solutions & training
>> Hadoop, Cascading, Mahout & Solr
>>
>>
>>
>>
>>

Hi Jun,

On May 25, 2012, at 11:51am, Jun Rao wrote:

> Ken,
>
> If you do ./sbt update and ./sbt package, you will find all the jars and
> ddls needed.

I see both scala-2.7.7 and scala-2.8.0 in paths when poking around for jars. E.g.

./project/boot/scala-2.7.7/lib/scala-compiler.jar
./core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar

I'm assuming this is something Ivy is creating, when resolving dependencies.

But why are there two versions being referenced?

> The core Kafka project only depends on zookeeper, zkclient and
> scala. However, other contribs depend on a lot of more stuff.

Sure, but those contribs should be built into their own jars, and thus have their own dependencies.

And (I assume) Zookeeper is only needed for testing, otherwise it's essentially a "provided" dependency - somebody has to make sure Zookeeper is running, but that's not a "must be on the classpath" dependency.

-- Ken
> On Fri, May 25, 2012 at 11:13 AM, Ken Krugler
> <[EMAIL PROTECTED]>wrote:
>
>> Hi Jun,
>>
>> On May 25, 2012, at 10:34am, Jun Rao wrote:
>>
>>> Ken,
>>>
>>> We tried to do a binary release and ended up going through 10 RCs, mostly
>>> to sort out the license of all jars. We gave up in the end. One of the
>>> biggest problems is that we have contribs for clients in other languages,
>>> which introduces dependent libraries in C#, go, etc whose license
>> agreement
>>> is not totally clear to us. If someone can help out on that, we'd be
>> happy
>>> to do a binary release.
>>
>> If you send me the list of dependencies that the Kafka jar has, I could
>> help out.
>>
>> From the test project I created, it looks like kafka-0.7.0.jar depends on:
>>
>> - scala-library.jar
>> - zkclient.jar
>>
>> So for Java users (who'd care about being able to pull kafka from a repo)
>> are there any other dependencies?
>>
>> Thanks,
>>
>> -- Ken
>>
>>
>>
>>> On Fri, May 25, 2012 at 10:16 AM, Ken Krugler
>>> <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hi Jun,
>>>>
>>>> Here's my +1 for publishing Kafka (jar and pom.xml with dependencies) to
>>>> some public repo.
>>>>
>>>> That makes it _much_ easier for a project to include Kafka.
>>>>
>>>> Note that it's pretty easy to get it into the Apache repo; publishing to
>>>> Maven central is a bit more challenging...
>>>>
>>>> -- Ken
>>>>
>>>> On May 25, 2012, at 9:46am, Jun Rao wrote:
>>>>
>>>>> Sam,
>>>>>
>>>>> Because of the overhead of certifying every dependent jar, we chose to
>> do
>>>>> only the source release, not the binary release for Kafka. This
>> creates a
>>>>> bit overhead for our users since they have to build the source to
>>>> generate
>>>>> the kafka jar.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]>
>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> Im writing a framework involving kafka consumers in  scala and Im
>> using
>>>>>> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
>>>>>> available in any  maven or ivey repositories?
>>>>>>
>>>>>>
>>>>>> Sam William
>>>>>> [EMAIL PROTECTED]
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --------------------------
>>>> Ken Krugler
>>>> http://www.scaleunlimited.com
>>>> custom big data solutions & training
>>>> Hadoop, Cascading, Mahout & Solr
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>> --------------------------
>> Ken Krugler
>> http://www.scaleunlimited.com
>> custom big data solutions & training
>> Hadoop, Cascading, Mahout & Solr
>>
>>
>>
>>
>>

--------------------------------------------
http://about.me/kkrugler
+1 530-210-6378
--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr

I think it is worth thinking about this, it would be good to hear thoughts
from people maintaining clients. I think this approach was the overwhelming
recommendation from people working on client code. People had good
experiences working with redis and other projects that followed this
pattern.

The real thing we need to do to enable a rich client ecosystem is to
simplify the consumer protocol to make writing a distributed, partitioned
consumer as simple as possible (ideally no zk interaction needed). I think
we might have bandwidth to focus on this after 0.8 is out the door.

-Jay

On Fri, May 25, 2012 at 11:59 AM, Evan chan <[EMAIL PROTECTED]> wrote:

> Jun,
>
> What if the contrib and other clients were split off into their own
> projects?  This would help achieve binary release for core Kafka, plus
> enable faster development of the other projects (for example, I could
> easily fork and improve on the ruby client without worrying about the other
> code...)
>
> -Evan
> Carry your candle, run to the darkness
> Seek out the helpless, deceived and poor
> Hold out your candle for all to see it
> Take your candle, and go light your world
>
>
>
> On May 25, 2012, at 10:34 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Ken,
> >
> > We tried to do a binary release and ended up going through 10 RCs, mostly
> > to sort out the license of all jars. We gave up in the end. One of the
> > biggest problems is that we have contribs for clients in other languages,
> > which introduces dependent libraries in C#, go, etc whose license
> agreement
> > is not totally clear to us. If someone can help out on that, we'd be
> happy
> > to do a binary release.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, May 25, 2012 at 10:16 AM, Ken Krugler
> > <[EMAIL PROTECTED]>wrote:
> >
> >> Hi Jun,
> >>
> >> Here's my +1 for publishing Kafka (jar and pom.xml with dependencies) to
> >> some public repo.
> >>
> >> That makes it _much_ easier for a project to include Kafka.
> >>
> >> Note that it's pretty easy to get it into the Apache repo; publishing to
> >> Maven central is a bit more challenging...
> >>
> >> -- Ken
> >>
> >> On May 25, 2012, at 9:46am, Jun Rao wrote:
> >>
> >>> Sam,
> >>>
> >>> Because of the overhead of certifying every dependent jar, we chose to
> do
> >>> only the source release, not the binary release for Kafka. This
> creates a
> >>> bit overhead for our users since they have to build the source to
> >> generate
> >>> the kafka jar.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]>
> >> wrote:
> >>>
> >>>> Hi,
> >>>> Im writing a framework involving kafka consumers in  scala and Im
> using
> >>>> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
> >>>> available in any  maven or ivey repositories?
> >>>>
> >>>>
> >>>> Sam William
> >>>> [EMAIL PROTECTED]
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> >> --------------------------
> >> Ken Krugler
> >> http://www.scaleunlimited.com
> >> custom big data solutions & training
> >> Hadoop, Cascading, Mahout & Solr
> >>
> >>
> >>
> >>
> >>
>

Evan,

Moving some of the contrib and clients out could make sense. One of the
problems is that current committers don't necessarily have the skill set to
review patches for those contribs. We could either move those contribs to
other repositories or make the contributor of those contribs committers.

Thanks,

Jun

On Fri, May 25, 2012 at 11:59 AM, Evan chan <[EMAIL PROTECTED]> wrote:

> Jun,
>
> What if the contrib and other clients were split off into their own
> projects?  This would help achieve binary release for core Kafka, plus
> enable faster development of the other projects (for example, I could
> easily fork and improve on the ruby client without worrying about the other
> code...)
>
> -Evan
> Carry your candle, run to the darkness
> Seek out the helpless, deceived and poor
> Hold out your candle for all to see it
> Take your candle, and go light your world
>
>
>
> On May 25, 2012, at 10:34 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Ken,
> >
> > We tried to do a binary release and ended up going through 10 RCs, mostly
> > to sort out the license of all jars. We gave up in the end. One of the
> > biggest problems is that we have contribs for clients in other languages,
> > which introduces dependent libraries in C#, go, etc whose license
> agreement
> > is not totally clear to us. If someone can help out on that, we'd be
> happy
> > to do a binary release.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, May 25, 2012 at 10:16 AM, Ken Krugler
> > <[EMAIL PROTECTED]>wrote:
> >
> >> Hi Jun,
> >>
> >> Here's my +1 for publishing Kafka (jar and pom.xml with dependencies) to
> >> some public repo.
> >>
> >> That makes it _much_ easier for a project to include Kafka.
> >>
> >> Note that it's pretty easy to get it into the Apache repo; publishing to
> >> Maven central is a bit more challenging...
> >>
> >> -- Ken
> >>
> >> On May 25, 2012, at 9:46am, Jun Rao wrote:
> >>
> >>> Sam,
> >>>
> >>> Because of the overhead of certifying every dependent jar, we chose to
> do
> >>> only the source release, not the binary release for Kafka. This
> creates a
> >>> bit overhead for our users since they have to build the source to
> >> generate
> >>> the kafka jar.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]>
> >> wrote:
> >>>
> >>>> Hi,
> >>>> Im writing a framework involving kafka consumers in  scala and Im
> using
> >>>> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
> >>>> available in any  maven or ivey repositories?
> >>>>
> >>>>
> >>>> Sam William
> >>>> [EMAIL PROTECTED]
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> >> --------------------------
> >> Ken Krugler
> >> http://www.scaleunlimited.com
> >> custom big data solutions & training
> >> Hadoop, Cascading, Mahout & Solr
> >>
> >>
> >>
> >>
> >>
>

Ken,

Kafka code depends on scala 2.8. Maybe sbt itself depends on scala 2.7.7.

Kafka requires ZK jar. In particular, our consumer logic depends on ZK.

Thanks,

Jun

On Fri, May 25, 2012 at 12:32 PM, Ken Krugler
<[EMAIL PROTECTED]>wrote:

> Hi Jun,
>
> On May 25, 2012, at 11:51am, Jun Rao wrote:
>
> > Ken,
> >
> > If you do ./sbt update and ./sbt package, you will find all the jars and
> > ddls needed.
>
> I see both scala-2.7.7 and scala-2.8.0 in paths when poking around for
> jars. E.g.
>
> ./project/boot/scala-2.7.7/lib/scala-compiler.jar
> ./core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar
>
> I'm assuming this is something Ivy is creating, when resolving
> dependencies.
>
> But why are there two versions being referenced?
>
> > The core Kafka project only depends on zookeeper, zkclient and
> > scala. However, other contribs depend on a lot of more stuff.
>
> Sure, but those contribs should be built into their own jars, and thus
> have their own dependencies.
>
> And (I assume) Zookeeper is only needed for testing, otherwise it's
> essentially a "provided" dependency - somebody has to make sure Zookeeper
> is running, but that's not a "must be on the classpath" dependency.
>
> -- Ken
>
>
> > On Fri, May 25, 2012 at 11:13 AM, Ken Krugler
> > <[EMAIL PROTECTED]>wrote:
> >
> >> Hi Jun,
> >>
> >> On May 25, 2012, at 10:34am, Jun Rao wrote:
> >>
> >>> Ken,
> >>>
> >>> We tried to do a binary release and ended up going through 10 RCs,
> mostly
> >>> to sort out the license of all jars. We gave up in the end. One of the
> >>> biggest problems is that we have contribs for clients in other
> languages,
> >>> which introduces dependent libraries in C#, go, etc whose license
> >> agreement
> >>> is not totally clear to us. If someone can help out on that, we'd be
> >> happy
> >>> to do a binary release.
> >>
> >> If you send me the list of dependencies that the Kafka jar has, I could
> >> help out.
> >>
> >> From the test project I created, it looks like kafka-0.7.0.jar depends
> on:
> >>
> >> - scala-library.jar
> >> - zkclient.jar
> >>
> >> So for Java users (who'd care about being able to pull kafka from a
> repo)
> >> are there any other dependencies?
> >>
> >> Thanks,
> >>
> >> -- Ken
> >>
> >>
> >>
> >>> On Fri, May 25, 2012 at 10:16 AM, Ken Krugler
> >>> <[EMAIL PROTECTED]>wrote:
> >>>
> >>>> Hi Jun,
> >>>>
> >>>> Here's my +1 for publishing Kafka (jar and pom.xml with dependencies)
> to
> >>>> some public repo.
> >>>>
> >>>> That makes it _much_ easier for a project to include Kafka.
> >>>>
> >>>> Note that it's pretty easy to get it into the Apache repo; publishing
> to
> >>>> Maven central is a bit more challenging...
> >>>>
> >>>> -- Ken
> >>>>
> >>>> On May 25, 2012, at 9:46am, Jun Rao wrote:
> >>>>
> >>>>> Sam,
> >>>>>
> >>>>> Because of the overhead of certifying every dependent jar, we chose
> to
> >> do
> >>>>> only the source release, not the binary release for Kafka. This
> >> creates a
> >>>>> bit overhead for our users since they have to build the source to
> >>>> generate
> >>>>> the kafka jar.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>> On Thu, May 24, 2012 at 10:34 PM, Sam William <[EMAIL PROTECTED]
> >
> >>>> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>> Im writing a framework involving kafka consumers in  scala and Im
> >> using
> >>>>>> sbt as the build tool.  Where do I get kafka-0.7.jar from ?  Is it
> >>>>>> available in any  maven or ivey repositories?
> >>>>>>
> >>>>>>
> >>>>>> Sam William
> >>>>>> [EMAIL PROTECTED]
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>> --------------------------
> >>>> Ken Krugler
> >>>> http://www.scaleunlimited.com
> >>>> custom big data solutions & training
> >>>> Hadoop, Cascading, Mahout & Solr
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> >> --------------------------
> >> Ken Krugler
> >> http://www.scaleunlimited.com
> >> custom big data solutions & training
> >> Hadoop, Cascading, Mahout & Solr
> >>
> >>
> >>
> >>
> >