Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Encryption in HDFS


Copy link to this message
-
Encryption in HDFS
Seonyeong Bak 2013-02-26, 05:10
Hello, I'm a university student.

I implemented AES and Triple DES with CompressionCodec in java cryptography
architecture (JCA)
The encryption is performed by a client node using Hadoop API.
Map tasks read blocks from HDFS and these blocks are decrypted by each map
tasks.
I tested my implementation with generic HDFS.
My cluster consists of 3 nodes (1 master node, 3 worker nodes) and each
machines have quad core processor (i7-2600) and 4GB memory.
A test input is 1TB text file which consists of 32 multiple text files (1
text file is 32GB)

I expected that the encryption takes much more time than generic HDFS.
The performance does not differ significantly.
The decryption step takes about 5-7% more than generic HDFS.
The encryption step takes about 20-30% more than generic HDFS because it is
implemented by single thread and executed by 1 client node.
So the encryption can get more performance.

May there be any error in my test?

I know there are several implementation for encryting files in HDFS.
Are these implementations enough to secure HDFS?

best regards,

seonpark

* Sorry for my bad english
+
java8964 java8964 2013-02-26, 19:52
+
Michael Segel 2013-02-27, 12:58
+
Mathias Herberts 2013-02-26, 06:43
+
Seonyeong Bak 2013-02-28, 15:28