Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Writing Unsolicited Messages to a Connected Netty Client


Copy link to this message
-
Re: Writing Unsolicited Messages to a Connected Netty Client
For certain kinds of data it would be useful to continuously stream data
from server to client (or vice-versa).  This can be represented as an Avro
array response or request where each array element triggers a callback at
the receiving end.  This likely requires an extension to the avro spec, but
is much more capable than a polling solution.  It is related to Comet in the
sense that the RPC request is long lived, but is effectively a sequence of
smaller inverse RPCs.  Poling in general has built-in race conditions for
many types of information exchange and should be avoided whenever such race
conditions exist.

For streaming large volumes of data, this would be much more efficient than
an individual RPC per item.  For example, if the RPC is "I need to know
every state change in X" polling is not an option, but streaming is.  If the
requirement is "I need to know when the next state change occurs, but do not
need to know all changes" polling is OK, and streaming may send too much
data.

On 1/20/12 11:25 AM, "Armin Garcia" <[EMAIL PROTECTED]> wrote:

> Hi James,
>
> I see your point. On a different NIO framework, I implemented exactly the same
> message handling procedure (ie message routing) you just described. I guess I
> was pushing the NettyTransceiver a bit beyond its intended scope.
>
> I'll take a look at the comet pattern and see what I can do with it.
>
> Again, thanks Shaun & James.
>
>     -Armin
>
>
> On Fri, Jan 20, 2012 at 10:15 AM, James Baldassari <[EMAIL PROTECTED]>
> wrote:
>> Hi Armin,
>>
>> First I'd like to explain why the server-initiated messages are problematic.
>> Allowing the server to send unsolicited messages back to the client may work
>> for some Transceiver implementations (possibly PHP), but this change would
>> not be compatible with NettyTransceiver.  When the NettyTransceiver receives
>> a message from the server, it needs to know which callback to invoke in order
>> to pass the message back correctly to the client.  There could be several
>> RPCs "in flight" concurrently, so one of NettyTransceiver's jobs is to match
>> up the response with the request that initiated it.  If the client didn't
>> initiate the RPC then NettyTransceiver won't know where to deliver the
>> message, unless there were some catch-all callback that would be invoked
>> whenever one of these "unsolicited" messages were received.  So although
>> you're probably only interested in the PHP client, allowing the server to
>> send these unsolicited messages would potentially break NettyTransceiver (and
>> possibly other implementations as well).
>>
>> Shaun's idea of having the client poll the server periodically would
>> definitely work.  What we want to do is have the client receive notifications
>> from the server as they become available on the server side, but we also
>> don't want the client to be polling with such a high frequency that a lot of
>> CPU and bandwidth resources are wasted.  I think we can get the best of both
>> worlds by copying the Comet pattern, i.e. the "long poll" but using the Avro
>> RPC layer instead of (or on top of) HTTP.  First we'll start with Shaun's
>> update listener interface:
>>
>> protocol WeatherUpdateListener {
>>   WeatherUpdate listenForUpdate();
>> }
>>
>> The PHP client would invoke this RPC against the server in a tight loop.  On
>> the server side, the RPC will block until there is an update that is ready to
>> be sent to the client.  When the client does receive an event from the server
>> (or some timeout occurs), the client will immediately send another poll to
>> the server and block until the next update is received.  In this way the
>> client will not be flooding the server with RPCs, but the client will also
>> get updates in a timely manner.
>>
>> See the following for more info about Comet:
>> http://www.javaworld.com/javaworld/jw-03-2008/jw-03-asynchhttp.html?page=6
>>
>> -James
>>
>>
>>
>> On Fri, Jan 20, 2012 at 12:44 PM, Armin Garcia <[EMAIL PROTECTED]>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB