Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Handling protocol versions

Devaraj Das 2012-07-31, 00:47
Jimmy Xiang 2012-07-31, 03:09
Ted Yu 2012-07-31, 03:11
Ted Yu 2012-07-31, 21:50
Stack 2012-08-01, 08:41
Todd Lipcon 2012-08-01, 18:04
Andrew Purtell 2012-08-01, 19:39
Copy link to this message
Re: Handling protocol versions
Responses inline..

> On Wed, Aug 1, 2012 at 11:04 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>> One possibility:
>> During the IPC handshake, we could send the full version string /
>> source checksum. Then, have a client-wide map which caches which
>> methods have been found to be supported or not supported for an
>> individual version. So, we don't need to maintain the mapping
>> ourselves, but we also wouldn't need to do the full retry every time.

Yeah this is what I was thinking as the alternate to the current approach of using VersionedProtocol.

>> A different idea would be to introduce a call like
>> "getServerCapabilities()" which returns a bitmap, and define a bit per
>> time that we add a new feature.
>> The advantage of these approaches vs a single increasing version
>> number is that we sometimes want to backport a new IPC to an older
>> version, but not backport all of the intervening IPCs. Having a bitmap
>> allows us to "pick and choose" on backports without having to pull in
>> a bunch of things we didn't necessarily want.

Good point.

>> On Wed, Aug 1, 2012 at 1:41 AM, Stack <[EMAIL PROTECTED]> wrote:
>>> On Tue, Jul 31, 2012 at 1:47 AM, Devaraj Das <[EMAIL PROTECTED]> wrote:
>>>> Wondering whether we should retain the VersionedProtocol now that we have protobuf implementation for most (all?) of the protocols. I think we still need the version checks and do them when we need to. Take this case:
>>>> 1. Protocol Foo has as one of the methods FooMethod(FooMethodRequest).
>>>> 2. Protocol Foo evolves over time, and the FooMethod(FooMethodRequest) now has a better implementation called FooMethod_improved(FooMethodRequest).
>>>> 3. HBase installations have happened with both the protocol implementations.
>>>> 4. Clients should be able to talk to both old and new servers (and invoke the newer implementation of FooMethod if the protocol implements it).
>>>> (4) is possible when the getProtocolVersion is implemented by the protocol at the server. The client could check what the version of the protocol was (assuming VersionedProtocol semantics where the protocol version number is upgraded for such significant changes) and depending on that invoke the appropriate method...
>>>> Having to map version-numbers of protocols to the methods-supported is probably arcane IMO but works..
>>>> The other approach (that wouldn't require the version#) is to do something like - On the client side, get the protocol methods supported at the server (and cache it) and then look this map up whenever needed to decide which method to invoke.
>>>> Any thoughts on whether we should invest time in the second approach yet?
>>> The VersionedProtocol w/ client being able to interrogate what methods
>>> a server supports strikes me as a facility that will be rarely used if
>>> at all and bringing it along, keeping up the directory of supported
>>> methods, will take a load of work on our part that we'll do less than
>>> perfectly so should it ever be needed, it won't work because we let it
>>> go stale.

Yeah, this won't be a common case. It'd (hopefully) be rare. The directory of methods would be the methods in the protocol-interface at the server that could be figured by invoking reflection (and hence staleness issue shouldn't happen).

>>> What do you reckon?
>>> The above painted scenario too is a little on the exotic side.  We can
>>> do something like Jimmy suggests in those rare cases we need to add a
>>> new method because there is insufficient wiggle-room w/i the
>>> particular PB method call (If we get into the issue Ted raises where
>>> we'd have to go back to the server twice because there is a third new
>>> method call, we're doing our API wrong).

Agree that the exception handling hack can be played here.. In general, having some solution around this might be really helpful *if* we get some API wrong (for e.g., indirect implication on memory by the API semantics) and we need to fix it without breaking compatibility.. In HDFS, listFile proved to be a memory killer for extremely large directories and people implemented the iterator version of the same.
The version in "hrpc" is the RPC version (as opposed to protocol version). I think that's orthogonal to this discussion..
As I answered above, and as Todd also says, it probably makes sense to have a client wide cache for protocol<->supported-methods .. and look up the cache when and if the client needs to decide between different versions of a method, or picking a new method, based on the server it is talking to...
Stack 2012-12-27, 20:05
Jimmy Xiang 2012-12-27, 21:13
Enis Söztutar 2012-12-28, 01:37
Stack 2012-12-28, 03:11
Stack 2013-01-02, 00:44
Elliott Clark 2013-01-02, 22:34
Stack 2013-01-07, 21:15
Devaraj Das 2012-12-28, 09:31
Stack 2012-12-28, 16:59
Devaraj Das 2012-12-28, 17:22