Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Authentication - Avro Source, Sink, RpcClient


Copy link to this message
-
Re: Authentication - Avro Source, Sink, RpcClient
Rudolf, wow interesting! So it's working, huh? Can you submit a patch?

Regards,
Mike

On Tuesday, February 5, 2013, Rakos, Rudolf wrote:

>   Yes, I think so.****
>
> ** **
>
> I had to make 4 small changes. ****
>
> Unfortunately I couldn’t subclass these classes, but I could reuse most of
> the code.****
>
> ** **
>
> **·         **First I added an “auth” configuration option to AvroSource
> and AvroSink, to be able to enable / disable authentication.****
>
> **·         **AvroSource#start()
>
> Responder responder = *new* SpecificResponder(AvroSourceProtocol.*class*,
> *this*);
> *if* (auth) {
>     responder.addRPCPlugin(*new* AuthenticatorRPCPlugin());
> }
>
> ****
>
> **·         **AvroSink#createConnection()
>
> Instead of using RpcClientFactory:
> client = RpcClientFactory.getInstance(clientProps);
>
> I create my own RpcClient (I didn’t want to modify RpcClientFactory):
> client = *new* AuthNettyAvroRpcClient(auth, ...);
>
> ****
>
> **·         **AuthNettyAvroRpcClient#connect(long,
> java.util.concurrent.TimeUnit)
>
> Instead of getting a client using the Transceiver:
> avroClient = SpecificRequestor.getClient(AvroSourceProtocol.Callback.*
> class*, transceiver);
>
> I get a client using a Requestor:
> SpecificRequestor requestor = *new*SpecificRequestor(AvroSourceProtocol.Callback.
> *class*, transceiver);
> *if* (auth) {
>     requestor.addRPCPlugin(*new* AuthenticatorRPCPlugin());
> }
> avroClient = SpecificRequestor.*getClient*(AvroSourceProtocol.Callback.*
> class*, requestor);****
>
> ** **
>
> The AuthenticatorRPCPlugin is an org.apache.avro.ipc.RPCPlugin with 3
> methods overriden to exchange the authentication data during the Avro
> handshake:****
>
> **1.       **clientStartConnect(org.apache.avro.ipc.RPCContext)
> This is called on the client, before the handshake.****
>
> **2.       **serverConnecting(org.apache.avro.ipc.RPCContext)
> This is called on the server. The connection will be aborted, if an
> exception is thrown here.****
>
> **3.       **clientFinishConnect(org.apache.avro.ipc.RPCContext)
> This is called on the client, after the handshake.****
>
> The org.apache.avro.ipc.RPCContext#getHandshakeRequest() returns an
> org.apache.avro.ipc.HandshakeRequest which contains a java.util.Map<java.lang.String,
> java.nio.ByteBuffer> where I could store the authentication data (
> org.apache.avro.ipc.HandshakeRequest#getMeta andorg.apache.avro.ipc.HandshakeRequest#
> setMeta methods).****
>
> ** **
>
> This might not be the nicest way to implement authentication, but this way
> it’s pretty much transparent to Flume.****
>
> I think it would be pretty easy to implement some kind of encryption too
> using the other org.apache.avro.ipc.RPCPlugin methods.****
>
> ** **
>
> Regards,****
>
> Rudolf****
>
> ** **
>
> *From:* Mike Percy [mailto:[EMAIL PROTECTED] <javascript:_e({}, 'cvml',
> '[EMAIL PROTECTED]');>]
> *Sent:* Tuesday, February 05, 2013 9:33 AM
> *To:* [EMAIL PROTECTED] <javascript:_e({}, 'cvml',
> '[EMAIL PROTECTED]');>
> *Subject:* Re: Authentication - Avro Source, Sink, RpcClient****
>
> ** **
>
> Rudolf did you try it? Any luck?****
>
> ** **
>
> Regards****
>
> Mike
>
> On Friday, January 25, 2013, Rakos, Rudolf wrote:****
>
> I think that maybe it’s possible to use Avro RPCPlugins to inject
> authentication data during the Avro handshake.****
>
>  ****
>
> Thanks for your help Mike.****
>
>  ****
>
> Regards,****
>
> Rudolf****
>
>  ****
>
> *From:* Mike Percy [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, January 23, 2013 9:16 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Authentication - Avro Source, Sink, RpcClient****
>
>  ****
>
> I agree that AvroSource/Sink SASL and Kerberos auth would be really
> useful. It would need some work at the Avro level, though.****
>
>  ****
>
> There is also the possibility of doing the same thing on top of Thrift, in
> which case it would require a brand new source/sink/client implementation
> but it wouldn't require any protocol work AFAICT.****