Another challenge of encrypt/decrypt is key management.
Can  you share how are this is handled in your implementation/research

2013/2/25 Seonyeong Bak <[EMAIL PROTECTED]>

> Hello, I'm a university student.
>
> I implemented AES and Triple DES with CompressionCodec in java
> cryptography architecture (JCA)
> The encryption is performed by a client node using Hadoop API.
> Map tasks read blocks from HDFS and these blocks are decrypted by each map
> tasks.
> I tested my implementation with generic HDFS.
> My cluster consists of 3 nodes (1 master node, 3 worker nodes) and each
> machines have quad core processor (i7-2600) and 4GB memory.
> A test input is 1TB text file which consists of 32 multiple text files (1
> text file is 32GB)
>
> I expected that the encryption takes much more time than generic HDFS.
> The performance does not differ significantly.
> The decryption step takes about 5-7% more than generic HDFS.
> The encryption step takes about 20-30% more than generic HDFS because it
> is implemented by single thread and executed by 1 client node.
> So the encryption can get more performance.
>
> May there be any error in my test?
>
> I know there are several implementation for encryting files in HDFS.
> Are these implementations enough to secure HDFS?
>
> best regards,
>
> seonpark
>
> * Sorry for my bad english
>
>
--
Have a Nice Day!
Lohit

The following JIRAs are related to your research:

HADOOP-9331: Hadoop crypto codec framework and crypto codec implementations<
https://issues.apache.org/jira/browse/hadoop-9331> and related sub-tasks

MAPREDUCE-5025: Key Distribution and Management for supporting crypto codec
in Map Reduce<https://issues.apache.org/jira/browse/mapreduce-5025> and
related JIRAs

On Mon, Feb 25, 2013 at 9:10 PM, Seonyeong Bak <[EMAIL PROTECTED]> wrote:

> Hello, I'm a university student.
>
> I implemented AES and Triple DES with CompressionCodec in java
> cryptography architecture (JCA)
> The encryption is performed by a client node using Hadoop API.
> Map tasks read blocks from HDFS and these blocks are decrypted by each map
> tasks.
> I tested my implementation with generic HDFS.
> My cluster consists of 3 nodes (1 master node, 3 worker nodes) and each
> machines have quad core processor (i7-2600) and 4GB memory.
> A test input is 1TB text file which consists of 32 multiple text files (1
> text file is 32GB)
>
> I expected that the encryption takes much more time than generic HDFS.
> The performance does not differ significantly.
> The decryption step takes about 5-7% more than generic HDFS.
> The encryption step takes about 20-30% more than generic HDFS because it
> is implemented by single thread and executed by 1 client node.
> So the encryption can get more performance.
>
> May there be any error in my test?
>
> I know there are several implementation for encryting files in HDFS.
> Are these implementations enough to secure HDFS?
>
> best regards,
>
> seonpark
>
> * Sorry for my bad english
>
>

I didn't handle a key distribution problem because I thought that this
problem is more difficult.
I simply hardcode a key into the code.

A challenge related to security are handled in
HADOOP-9331<https://issues.apache.org/jira/browse/HADOOP-9331>,
MAPREDUCE-5025 <https://issues.apache.org/jira/browse/MAPREDUCE-5025>, and
so on.

Most recent crypto libraries use the special instructions on Intel
processors.

See for instance:
http://software.intel.com/en-us/articles/intel-advanced-encryption-standard-aes-instructions-set
On Mon, Feb 25, 2013 at 9:10 PM, Seonyeong Bak <[EMAIL PROTECTED]> wrote:

> Hello, I'm a university student.
>
> I implemented AES and Triple DES with CompressionCodec in java
> cryptography architecture (JCA)
> The encryption is performed by a client node using Hadoop API.
> Map tasks read blocks from HDFS and these blocks are decrypted by each map
> tasks.
> I tested my implementation with generic HDFS.
> My cluster consists of 3 nodes (1 master node, 3 worker nodes) and each
> machines have quad core processor (i7-2600) and 4GB memory.
> A test input is 1TB text file which consists of 32 multiple text files (1
> text file is 32GB)
>
> I expected that the encryption takes much more time than generic HDFS.
> The performance does not differ significantly.
> The decryption step takes about 5-7% more than generic HDFS.
> The encryption step takes about 20-30% more than generic HDFS because it
> is implemented by single thread and executed by 1 client node.
> So the encryption can get more performance.
>
> May there be any error in my test?
>
> I know there are several implementation for encryting files in HDFS.
> Are these implementations enough to secure HDFS?
>
> best regards,
>
> seonpark
>
> * Sorry for my bad english
>
>

Thank you so much for all your comments. :)

Excellent!

On 02/25/2013 10:43 PM, Mathias Herberts wrote:
> Encryption without proper key management only addresses the 'stolen
> hard drive' problem.
>
> So far I have not found 100% satisfactory solutions to this hard
> problem. I've written OSS (Open Secret Server) partly to address this
> problem in Pig, i.e. accessing encrypted data without embedding key
> info into the job description file. Proper encrypted data handling
> implies striict code review though, as in the case of Pig databags are
> spillable and you could end up with unencrypted data stored on disk
> without intent.
>
> OSS http://github.com/hbs/oss and the Pig specific code:
> https://github.com/hbs/oss/blob/master/src/main/java/com/geoxp/oss/pig/PigSecretStore.java
>
> On Tue, Feb 26, 2013 at 6:33 AM, Seonyeong Bak <[EMAIL PROTECTED]> wrote:
>> I didn't handle a key distribution problem because I thought that this
>> problem is more difficult.
>> I simply hardcode a key into the code.
>>
>> A challenge related to security are handled in HADOOP-9331, MAPREDUCE-5025,
>> and so on.