Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Securing Kafka


+
Fergal Somers 2013-04-23, 17:03
+
Jason Rosenberg 2013-04-23, 18:03
+
Matt Wise 2013-04-23, 18:29
Copy link to this message
-
Re: Securing Kafka
Chris Curtin 2013-04-23, 18:31
Also keep in mind that anything done at the transport (SSL for example)
layer won't solve your 'at rest' problems.

All messages are written to disk, so unless the broker does some encryption
logic you haven't solved the data visibility issues.

I also think this should be a producer/consumer problem not a Broker. Keep
the Brokers as fast as possible (thus NIO/kernel space activities etc.) and
push the cost to the producers and consumers.

Chris
On Tue, Apr 23, 2013 at 2:02 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Yes,
>
> I think encryption at the message level is a workable solution, as long as
> you don't care about exposing the meta data that goes with it (e.g. topic
> names, kafka broker/zk server locations, etc.).
>
> Jason
>
>
> On Tue, Apr 23, 2013 at 10:02 AM, Fergal Somers
> <[EMAIL PROTECTED]>wrote:
>
> > Hi
> >
> > We are planning to use Kafka, but like others on this list we have a need
> > to be able to secure communication. The approaches people have suggested
> on
> > this list are:
> >
> >    - Encrypt the messages at the producer (e.g
> >
> >
> http://search-hadoop.com/m/1AfXKcZIk52/message+encryption&subj=Re+Secure+communication
> >    )
> >    - Add SSL to Kafka protocol -
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201304.mbox/ajax/%3CCAA%2BBczQ_dMXUTNndSu4d%2B6aRo%3DSLiFa4iGMu_78OWKub_CTScw%40mail.gmail.com%3E
> >
> > Adding SSL support to Kafka, probably means adding SSLEngine support the
> > the nio socket handling (
> > https://groups.google.com/forum/#!msg/kafka-dev/ZmJrB_plu1I/_9cmGlLCSVEJ
> ).
> > I don't think there are any immediate plans to provide this, but it's
> > potentially something that Kafka would support in the future?
> >
> > In theory this is something we could look at, but we would need to go
> > further. We also need to separate producers from consumers. The aim would
> > be to ensure that a Kafka producer couldn't also act as a consumer.
> > Essentially producers can write to Kafka, but not read.
> >
> > From looking at the Kafka source, achieving producer/consumer separation
> > looks to me like it would be quite a change to the Kafka server (0.7). So
> > are there any plans in the (near) future in this area (producer /
> consumer
> > separation) ?
> >
> > Message encryption (at the application layer) would allow us to achieve
> > both aims of securing communication and separating consumers from
> > producers. Producers get the public cert (so they can encrypt messages as
> > they place them on the bus). Only consumers get the private cert - so
> only
> > they can decrypt messages consumed. This seems like something we can do
> > ourselves - I just wanted to sanity check the approach with this group.
> >
> > Cheers,
> >
> > Fergal.
> >
>

 
+
Fergal Somers 2013-04-26, 10:57
+
Calvin Lei 2013-08-29, 17:03