Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Authentication - Avro Source, Sink, RpcClient


+
Rakos, Rudolf 2013-01-23, 11:45
+
Mike Percy 2013-01-23, 20:16
+
Rakos, Rudolf 2013-01-25, 14:07
+
Mike Percy 2013-02-05, 08:33
+
Rakos, Rudolf 2013-02-05, 10:50
Copy link to this message
-
Re: Authentication - Avro Source, Sink, RpcClient
Rudolf, wow interesting! So it's working, huh? Can you submit a patch?

Regards,
Mike

On Tuesday, February 5, 2013, Rakos, Rudolf wrote:

>   Yes, I think so.****
>
> ** **
>
> I had to make 4 small changes. ****
>
> Unfortunately I couldn’t subclass these classes, but I could reuse most of
> the code.****
>
> ** **
>
> **·         **First I added an “auth” configuration option to AvroSource
> and AvroSink, to be able to enable / disable authentication.****
>
> **·         **AvroSource#start()
>
> Responder responder = *new* SpecificResponder(AvroSourceProtocol.*class*,
> *this*);
> *if* (auth) {
>     responder.addRPCPlugin(*new* AuthenticatorRPCPlugin());
> }
>
> ****
>
> **·         **AvroSink#createConnection()
>
> Instead of using RpcClientFactory:
> client = RpcClientFactory.getInstance(clientProps);
>
> I create my own RpcClient (I didn’t want to modify RpcClientFactory):
> client = *new* AuthNettyAvroRpcClient(auth, ...);
>
> ****
>
> **·         **AuthNettyAvroRpcClient#connect(long,
> java.util.concurrent.TimeUnit)
>
> Instead of getting a client using the Transceiver:
> avroClient = SpecificRequestor.getClient(AvroSourceProtocol.Callback.*
> class*, transceiver);
>
> I get a client using a Requestor:
> SpecificRequestor requestor = *new*SpecificRequestor(AvroSourceProtocol.Callback.
> *class*, transceiver);
> *if* (auth) {
>     requestor.addRPCPlugin(*new* AuthenticatorRPCPlugin());
> }
> avroClient = SpecificRequestor.*getClient*(AvroSourceProtocol.Callback.*
> class*, requestor);****
>
> ** **
>
> The AuthenticatorRPCPlugin is an org.apache.avro.ipc.RPCPlugin with 3
> methods overriden to exchange the authentication data during the Avro
> handshake:****
>
> **1.       **clientStartConnect(org.apache.avro.ipc.RPCContext)
> This is called on the client, before the handshake.****
>
> **2.       **serverConnecting(org.apache.avro.ipc.RPCContext)
> This is called on the server. The connection will be aborted, if an
> exception is thrown here.****
>
> **3.       **clientFinishConnect(org.apache.avro.ipc.RPCContext)
> This is called on the client, after the handshake.****
>
> The org.apache.avro.ipc.RPCContext#getHandshakeRequest() returns an
> org.apache.avro.ipc.HandshakeRequest which contains a java.util.Map<java.lang.String,
> java.nio.ByteBuffer> where I could store the authentication data (
> org.apache.avro.ipc.HandshakeRequest#getMeta andorg.apache.avro.ipc.HandshakeRequest#
> setMeta methods).****
>
> ** **
>
> This might not be the nicest way to implement authentication, but this way
> it’s pretty much transparent to Flume.****
>
> I think it would be pretty easy to implement some kind of encryption too
> using the other org.apache.avro.ipc.RPCPlugin methods.****
>
> ** **
>
> Regards,****
>
> Rudolf****
>
> ** **
>
> *From:* Mike Percy [mailto:[EMAIL PROTECTED] <javascript:_e({}, 'cvml',
> '[EMAIL PROTECTED]');>]
> *Sent:* Tuesday, February 05, 2013 9:33 AM
> *To:* [EMAIL PROTECTED] <javascript:_e({}, 'cvml',
> '[EMAIL PROTECTED]');>
> *Subject:* Re: Authentication - Avro Source, Sink, RpcClient****
>
> ** **
>
> Rudolf did you try it? Any luck?****
>
> ** **
>
> Regards****
>
> Mike
>
> On Friday, January 25, 2013, Rakos, Rudolf wrote:****
>
> I think that maybe it’s possible to use Avro RPCPlugins to inject
> authentication data during the Avro handshake.****
>
>  ****
>
> Thanks for your help Mike.****
>
>  ****
>
> Regards,****
>
> Rudolf****
>
>  ****
>
> *From:* Mike Percy [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, January 23, 2013 9:16 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Authentication - Avro Source, Sink, RpcClient****
>
>  ****
>
> I agree that AvroSource/Sink SASL and Kerberos auth would be really
> useful. It would need some work at the Avro level, though.****
>
>  ****
>
> There is also the possibility of doing the same thing on top of Thrift, in
> which case it would require a brand new source/sink/client implementation
> but it wouldn't require any protocol work AFAICT.****
+
Rakos, Rudolf 2013-02-07, 09:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB