Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - Handling protocol versions


+
Devaraj Das 2012-07-31, 00:47
+
Jimmy Xiang 2012-07-31, 03:09
+
Ted Yu 2012-07-31, 03:11
+
Ted Yu 2012-07-31, 21:50
+
Stack 2012-08-01, 08:41
+
Todd Lipcon 2012-08-01, 18:04
+
Andrew Purtell 2012-08-01, 19:39
+
Devaraj Das 2012-08-03, 18:40
Copy link to this message
-
Re: Handling protocol versions
Stack 2012-12-27, 20:05
So, picking up this thread again because I'm working on
https://issues.apache.org/jira/browse/HBASE-6521 "
Address the handling of multiple versions of a protocol"Address the
handling of multiple versions of a protocol", the original question was
 two-fold as I read it.

1. Should we keep VersionedProtocol.
2. How does a client figure if a server supports a particular capability

On question 1:

VersionedProtocol [1] does two things.  It returns the server version of
the protocol and separately, a "ProtocolSignature" Writable which allows
you get a 'hash' of the server's protocol method signatures.   There is an
implication that the server will give out different versions of the
protocol dependent on what version the client volunteers (not the case) and
it is implied that the client does something with these method hash
signatures.  It doesn't.

So, VP is a Writable that returns Writables we don't make use of implying a
functionality unrealized.

Thats how I read it.  Objections? [3]

It sounds like at least ProtocolSignature can go.  If we did want to go the
route ProtocolSignature implies, we should probably do the native protobuf
thing and make use of ServiceDescriptors, protobuf descriptions of what a
protobuf Service exposes [2].

That leaves the VPs return of the server protocol version as all that
remains 'useful'.

But is it? Is version going to be useful going forward?  If we lean on
version, clients will have to keep a registry of versions to available
methods.  Or ask the server what it has and somehow sort though the return
to figure what it can and cannot make sense of by method.  Sounds like a
bunch of work.

At a minimum, VP will have to be protobuf'd so it is going to have to
change.  And we should probably add a bit more info to the return since we
are going to the trouble of an RPC anyways.

This serves as a lead in to question 2:

Protobuf as is helps in the case where an ipc takes an extra parameter or
adds extra info to the return; the majority of the evolutions that will be
happening in the ipc interface.  But what to do about the scenario Devaraj
outlines at the head of the thread where we have shipped a method that
causes the server to OOME in production or we add a method to the server
that runs ten times faster than the old one?  Or probably more likely, the
server has a whole new 'feature' (as Todd calls it) orthogonal to the set
the protocol version implies?  How does the client figure the new feature
is available?

We could have the client try the invocation -- as Jimmy suggests -- and if
it fails, register the fail in a client-wide map so we avoid retrying on
each invocation (We should just do this anyways).  The client could go back
to the server and do the above suggested query of server capabilities and
then adjust the call accordingly or since we are doing an ipc setup call
anyways, we could have the server return the list of capabilities at this
time.  The client could cache what is available or not and just ask the
server when convenient for it.

Using the bitmap shorthand describing what is available seems like it would
be less work to do than implementing protobuf service
description/interrogation and then dynamically composing method calls.

Proposal:

+ Remove VersionedProtocol and SignatureProtocol
+ Instead of VP, add a new Interface called VersionedService or probably
better, ProtocolDescriptor, that all RPC Protocols implement.  It has
methods (getDescriptor) to return a pb Message that has the server version
of the protocol and a bitmap of feature's the server implements.  This is
the call we will make when we set up the ipc proxy.  Clients can cache the
result.  Every time we change a Service/Protocol, we set a particular bit
in the Service/Protocol bitmap.  This new Interface might also return the
long form pb ServiceDescriptors (the pb getDescriptorForType from Service
Interface).  It could be useful debugging.

What you lot think?

St.Ack

1.
http://svn.apache.org/viewvc/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/VersionedProtocol.java?view=markup
2.
https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/Service
3. We have VP and PS because, as I understand it, we once that we would
support choosing between protocol and protocol versions and that we'd
support both protobufs and Writables.  This is no longer an wanted.
On Fri, Aug 3, 2012 at 11:40 AM, Devaraj Das <[EMAIL PROTECTED]> wrote:

+
Jimmy Xiang 2012-12-27, 21:13
+
Enis Söztutar 2012-12-28, 01:37
+
Stack 2012-12-28, 03:11
+
Stack 2013-01-02, 00:44
+
Elliott Clark 2013-01-02, 22:34
+
Stack 2013-01-07, 21:15
+
Devaraj Das 2012-12-28, 09:31
+
Stack 2012-12-28, 16:59
+
Devaraj Das 2012-12-28, 17:22