I'm considering the use of Kafka in the rewrite of a big legacy product. A good chunk of the back end code is going to be written in C++ (large in memory data-structures). The two possible options available to me for clients appear to be:
The problem is that librdkafka currently only works on Linux due to the use of the AF_NETLINK API, and thread local storage. There may be other issues, but I just started playing with it today and that's what I've discovered thus far.
kafka-cpp is incomplete (no consumer) and it looks unused.
For either I would need to hop in and do some significant work. Is there any client I'm missing that can shorten my path?
If I adopt one of these projects (lets say kafka-cpp) am I better off implementing the 0.8 protocol? I'de like to have something running in staging a couple months from now. How far out is 0.8?
I have a working copy of kafka 0.8 producer in c++. I have not yet published it on github, since I did not get time to clean it up properly. If you wait till the weekend, I can clean up and share the repo. After that you can either use it or modify and use it.
Thanks, Rohit On Thu, Mar 28, 2013 at 7:38 AM, Magnus Edenhill <[EMAIL PROTECTED]> wrote:
I'm on OSX. It's not going to production on OSX, but I have a requirement that the entire stack be deployable on a developer's machine. OSX does have TLS, but not the local variable __thread. It would need to be reworked to use the pthreads API. That by itself isn't much work it's just I don't know what else is going to need to be fixed. On Thu, Mar 28, 2013 at 7:38 AM, Magnus Edenhill <[EMAIL PROTECTED]> wrote:
At LinkedIn, we are also building a native C producer client for 0.8. It uses non-blocking socket I/O to improve the producer throughput. We plan to open source it when it's fully tested, hopefully in a couple of months.
On Wed, Mar 27, 2013 at 8:48 PM, Matthew Stump <[EMAIL PROTECTED]>wrote:
Yes - we would be interested in doing that. I have been spending most of my time over the past couple weeks on the C++ library (currently, only for the producer). It is reasonably stable, although it has not been tried and tested in production.
I can start with publishing a wiki describing the design/implementation and list out future work that others can contribute to if there is interest, but I'll quickly summarize some of its goals here: - Use non-blocking I/O for sending requests/receiving responses. This mitigates the request-response RTT latency that the 0.8 protocol introduces (if acks != 0). - C-compliant API, so that other languages e.g., Ruby/Python can wrap the library. I understand there are people working on 0.8 clients in those languages so this is just an alternative approach. - Non-blocking variants for all operations in the API: there are some use-cases where blocking in the user-thread is unacceptable. - Keep third-party libraries to a minimum to make porting to other platforms easier.
There are two options in pursuing this: 1 - Contribute it as part of the Kafka project. i.e., re-introduce the clients directory. We removed it because none of the committers were original authors of any of the non-Java clients and did not have the bandwidth/background to review patches for those clients. 2 - Maintain it externally. I was going to also suggest "sub-project" as a third option, but I heard from Jakob that it is no longer encouraged by the board and a client by itself would not be a substantial sub-project.
If we want to do (1), we would obviously go through the standard contribution process: i.e., create a jira for it and provide a patch that people can review; and other implementations will also be considered if contributors provide patches.
(2) is also fine, although one benefit with (1) is that it would be reasonable to use the existing infrastructure (mailing lists, issue tracking) for the Kafka project.
On Mon, Apr 1, 2013 at 8:22 AM, mrevilgnome <[EMAIL PROTECTED]> wrote:
I'd be very interested in road testing the 0.8 C client, and contributing if I can be of use. I am building a system which has a particularly challenging throughput / latency requirement. The current 0.7 platform works to a point, but as we ramp up we are struggling to hit our perf targets in both producer and consumer latency, particularly in the tails. We were always pushing what 0.7 can do, I think we have found the limits and the ability for 0.8 to drop latencies to a few millis will be a massive boost. So, I would favor option 1 specifically for the C client. I could see why a client set up as a hobby project and maintained as such would lag the main code, but I expect the C stuff to be keenly contributed to and worth managing as key code. Is it possible to get my hands on something beta? Thanks
The contents of this email including any attachments are confidential. If you have received this email in error, please advise the sender by return email and delete this email. Any unauthorised use of the contents of the email is prohibited and you must not disseminate, copy or distribute the message or use the information contained in the email or its attachments in any way.
The views or opinions expressed are the author's own and may not reflect the views or opinions of Tibra. Tibra does not guarantee the integrity of any emails or attached files. E-mails may be interfered with, may contain computer viruses or other defects. Under no circumstances do we accept liability for any loss or damage which may result from your receipt of this message or any attachments.
On 3 April 2013 05:59, Joel Koshy <[EMAIL PROTECTED]> wrote:
anand nalya 2013-05-17, 10:35
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext