Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Review Request 23208: Patch for KAFKA-1512


Copy link to this message
-
Re: Review Request 23208: Patch for KAFKA-1512

I think what I have may be right. My understanding is that
 - InetAddress is an IP address
 - SocketAddress, InetSocketAddress are host/port pairs (basically)
getInetAddress gets the remote ip address whereas getRemoteAddress gets the InetSocketAddress.

This is sort of crazy naming, but that is my understanding.

My thought was that it was better to work with InetAddress rather than host because host can be ambiguously defined (e.g. app301, app301.prod, app301.prod.linkedin.com, etc).

I think avg is not useful, and will be misleading in the case of a leak (e.g. one host with many, many connections). I think max would be useful, however it is a bit of a pain. The three options I could think of:
- add a gauge that iterates over all hosts in the map and computes the max. In the common case this will be fine, but it is O(N) in the number of hosts. So if you have a ton of connections and a low poll time for your metrics (some people use 1 sec) then you will spend something like 30% of one cpu core counting connections while holding the quota lock. In general I think we have tried to avoid operations that iterate over all hosts or connections to ensure the socket server scales to lots of connections without slowing down.
- Add a SortedSet on (count, ip) that gets adjusted on inc and dec. This is correct and O(1)ish but is just complicated.
- Implement a broken max that only goes up. This would be easily but annoying.

See above.

See above.

This is a really good point. I don't care too much about this option as it is going to be rarely used. What I think is more important is that we fix the option in the new producer/consumer before we do an official release.

Technically I think there is a race condition and either the exception or the read with -1 is possible, so I think you need both.
- Jay
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23208/#review47362
On July 3, 2014, 10:18 p.m., Jay Kreps wrote: