Hello developers !

I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:

"Hadoop Security Design: 
An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"

I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.

Best regards.
Thomas 

Hello Thomas,

you can look into this project. This is exactly what you are doing, but at
a larger scale.
https://github.com/intel-hadoop/project-rhino/

Hope this helps,

Regards,
Panshul
On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <[EMAIL PROTECTED]> wrote:

> Hello developers !
>
> I'm a student at the french university "Ensimag" and currently doing my
> master research on "Software security". Interested by cloud computing, I
> chose for subject : "Secure hadoop cluster inter-cloud environment".
> My idea is to develop a framework in order to improve the security of the
> Hadoop cluster running on the cloud of my uni. I have started by checking
> the "Hadoop research projects" proposed  on Hadoop Wiki and the following
> subject fits with mine:
>
> "Hadoop Security Design:
> An end-to-end proposal for how to support authentication and client side
> data encryption/decryption, so that large data sets can be stored in a
> public HDFS and only jobs launched by authenticated users can map-reduce or
> browse the data"
>
> I would like to know if there are already some developers on it so we can
> discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and
> cloud cumputing so if would be really great if you had some advices or
> hints for my research.
>
> Best regards.
> Thomas
>

--
Regards,
Ouch Whisper
010101010101

Hi Thomas -

I think that you need articulate the problems that you want to solve for
your university environment.
The subject that you chose indicates "inter-cloud environment" - so
depending on the inter-cloud problems that currently exist for your
environment there may be interesting work from the Rhino effort or with
Knox.

It seems that you are leaning toward data protection and encryption as a
solution to some problem within your stated problem subject.
I'd be interested in the usecase that you are addressing with it that is
"inter-cloud".
Another family of issues that would be interesting in the inter-cloud space
would be various identity federation issues across clouds.

@Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox
is related to it?
If so, Knox is not directly related to HttpFS though it will leverage
lessons learned and hopefully the experience of those involved.
The Knox gateway is more transparent and committed to serving REST APIs to
numerous Hadoop services rather than just HDFS.
The pluggable providers of Knox gateway will also facilitate easier
integration with customer's identity infrastructure in on-prem and cloud
provider environments.

Hope that helps to draw the distinction between Knox and HttpFS.

thanks,

--larry

On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <[EMAIL PROTECTED]>wrote:

> Is this in any way related to GatewayFS?
> I am also curious whether any one knows of plans to incorporate
> homomorphic encryption or secure multiparty into the rhino effort.
> C
>
> On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>
> I am not sure if you guys have heard it or not
>
> HortonWorks is in process to incubate a new apache project called Knox for
> hadoop security.
> More on this you can look at
>
> http://hortonworks.com/blog/introducing-knox-hadoop-security/
>
> http://wiki.apache.org/incubator/knox
>
>
> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <[EMAIL PROTECTED]> wrote:
>
>> Thank you very much Panshul, I'll take a look.
>>
>> Thomas.
>>
>>   ------------------------------
>> *De :* Panshul Whisper <[EMAIL PROTECTED]>
>> *À :* [EMAIL PROTECTED]; Thomas Nguy <[EMAIL PROTECTED]>
>> *Envoyé le :* Mercredi 27 février 2013 13h53
>> *Objet :* Re: Work on research project "Hadoop Security Design"
>>
>> Hello Thomas,
>>
>> you can look into this project. This is exactly what you are doing, but
>> at a larger scale.
>> https://github.com/intel-hadoop/project-rhino/
>>
>> Hope this helps,
>>
>> Regards,
>> Panshul
>>
>>
>> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <[EMAIL PROTECTED]>wrote:
>>
>> Hello developers !
>>
>> I'm a student at the french university "Ensimag" and currently doing my
>> master research on "Software security". Interested by cloud computing, I
>> chose for subject : "Secure hadoop cluster inter-cloud environment".
>> My idea is to develop a framework in order to improve the security of the
>> Hadoop cluster running on the cloud of my uni. I have started by
>> checking the "Hadoop research projects" proposed  on Hadoop Wiki and the
>> following subject fits with mine:
>>
>> "Hadoop Security Design:
>>  An end-to-end proposal for how to support authentication and client
>> side data encryption/decryption, so that large data sets can be stored in a
>> public HDFS and only jobs launched by authenticated users can map-reduce or
>> browse the data"
>>
>> I would like to know if there are already some developers on it so we can
>> discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and
>> cloud cumputing so if would be really great if you had some advices or
>> hints for my research.
>>
>> Best regards.
>> Thomas
>>
>>
>>
>>
>> --
>> Regards,
>> Ouch Whisper
>> 010101010101
>>
>>
>>
>
>
> --
> Nitin Pawar
>
>
>

Thomas,
Interesting distinctions articulated, thanks.
I knew of an effort called GarewayFS that had inter-cluster secure federation as one goal, there are some similarities to Knox.
C

On Feb 27, 2013, at 10:04 AM, Larry McCay <[EMAIL PROTECTED]> wrote:

> Hi Thomas -
>
> I think that you need articulate the problems that you want to solve for your university environment.
> The subject that you chose indicates "inter-cloud environment" - so depending on the inter-cloud problems that currently exist for your environment there may be interesting work from the Rhino effort or with Knox.
>
> It seems that you are leaning toward data protection and encryption as a solution to some problem within your stated problem subject.
> I'd be interested in the usecase that you are addressing with it that is "inter-cloud".
> Another family of issues that would be interesting in the inter-cloud space would be various identity federation issues across clouds.
>
> @Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox is related to it?
> If so, Knox is not directly related to HttpFS though it will leverage lessons learned and hopefully the experience of those involved.
> The Knox gateway is more transparent and committed to serving REST APIs to numerous Hadoop services rather than just HDFS.
> The pluggable providers of Knox gateway will also facilitate easier integration with customer's identity infrastructure in on-prem and cloud provider environments.
>
> Hope that helps to draw the distinction between Knox and HttpFS.
>
> thanks,
>
> --larry
>
> On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <[EMAIL PROTECTED]> wrote:
>> Is this in any way related to GatewayFS?
>> I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
>> C
>>
>> On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>>
>>> I am not sure if you guys have heard it or not
>>>
>>> HortonWorks is in process to incubate a new apache project called Knox for hadoop security.
>>> More on this you can look at
>>>
>>> http://hortonworks.com/blog/introducing-knox-hadoop-security/
>>>
>>> http://wiki.apache.org/incubator/knox
>>>
>>>
>>> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <[EMAIL PROTECTED]> wrote:
>>>> Thank you very much Panshul, I'll take a look.
>>>>
>>>> Thomas.
>>>>
>>>> De : Panshul Whisper <[EMAIL PROTECTED]>
>>>> À : [EMAIL PROTECTED]; Thomas Nguy <[EMAIL PROTECTED]>
>>>> Envoyé le : Mercredi 27 février 2013 13h53
>>>> Objet : Re: Work on research project "Hadoop Security Design"
>>>>
>>>> Hello Thomas,
>>>>
>>>> you can look into this project. This is exactly what you are doing, but at a larger scale.
>>>> https://github.com/intel-hadoop/project-rhino/
>>>>
>>>> Hope this helps,
>>>>
>>>> Regards,
>>>> Panshul
>>>>
>>>>
>>>> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <[EMAIL PROTECTED]> wrote:
>>>> Hello developers !
>>>>
>>>> I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
>>>> My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
>>>>
>>>> "Hadoop Security Design:
>>>> An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
>>>>
>>>> I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
>>>>
>>>> Best regards.

Thank you everyone for your answer. It gives me a lot of paths for reflection.

Thanks Larry, I'll have to dig more about the "inter-cloud" system my uni is using.

Thomas
________________________________
 De : Charles Earl <[EMAIL PROTECTED]>
À : "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc : "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Envoyé le : Mercredi 27 février 2013 16h09
Objet : Re: Work on research project "Hadoop Security Design"
 

Thomas, 
Interesting distinctions articulated, thanks.
I knew of an effort called GarewayFS that had inter-cluster secure federation as one goal, there are some similarities to Knox.
C

On Feb 27, 2013, at 10:04 AM, Larry McCay <[EMAIL PROTECTED]> wrote:
Hi Thomas -
>
>
>I think that you need articulate the problems that you want to solve for your university environment.
>The subject that you chose indicates "inter-cloud environment" - so depending on the inter-cloud problems that currently exist for your environment there may be interesting work from the Rhino effort or with Knox.
>
>
>It seems that you are leaning toward data protection and encryption as a solution to some problem within your stated problem subject.
>I'd be interested in the usecase that you are addressing with it that is "inter-cloud".
>Another family of issues that would be interesting in the inter-cloud space would be various identity federation issues across clouds.
>
>
>@Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox is related to it?
>If so, Knox is not directly related to HttpFS though it will leverage lessons learned and hopefully the experience of those involved.
>The Knox gateway is more transparent and committed to serving REST APIs to numerous Hadoop services rather than just HDFS.
>The pluggable providers of Knox gateway will also facilitate easier integration with customer's identity infrastructure in on-prem and cloud provider environments.
>
>
>Hope that helps to draw the distinction between Knox and HttpFS.
>
>thanks,
>
>
>--larry
>
>
>On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <[EMAIL PROTECTED]> wrote:
>
>Is this in any way related to GatewayFS?
>>I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
>>C
>>
>>
>>On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>>
>>I am not sure if you guys have heard it or not 
>>>
>>>
>>>HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
>>>More on this you can look at 
>>>
>>>
>>>http://hortonworks.com/blog/introducing-knox-hadoop-security/
>>>
>>>
>>>
>>>http://wiki.apache.org/incubator/knox
>>>
>>>
>>>
>>>
>>>On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <[EMAIL PROTECTED]> wrote:
>>>
>>>Thank you very much Panshul, I'll take a look.
>>>>
>>>>
>>>>Thomas.
>>>>
>>>>
>>>>
>>>>________________________________
>>>> De : Panshul Whisper <[EMAIL PROTECTED]>
>>>>À : [EMAIL PROTECTED]; Thomas Nguy <[EMAIL PROTECTED]>
>>>>Envoyé le : Mercredi 27 février 2013 13h53
>>>>Objet : Re: Work on research project "Hadoop Security Design"
>>>>
>>>>
>>>>Hello Thomas,
>>>>
>>>>
>>>>you can look into this project. This is exactly what you are doing, but at a larger scale. 
>>>>https://github.com/intel-hadoop/project-rhino/
>>>>
>>>>
>>>>
>>>>Hope this helps,
>>>>
>>>>
>>>>Regards,
>>>>Panshul
>>>>
>>>>
>>>>
>>>>On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <[EMAIL PROTECTED]> wrote:
>>>>
>>>>Hello developers !
>>>>>
>>>>>
>>>>>I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
>>>>>My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
>>>>

How to extract data from HDFS file system to our sytem? are there any
commands available?

>
>
regards
poonam

please start a new thread for your questions poonam

if your data is in plain files then you can do bin/hadoop dfs -cat file/path

you can copy the file to local  with bin/hadoop fs -copyToLocal
[-ignorecrc] [-crc] URI <localdst>
On Thu, Feb 28, 2013 at 12:57 PM, POONAM GHULI <[EMAIL PROTECTED]>wrote:

>
> How to extract data from HDFS file system to our sytem? are there any
> commands available?
>
>>
>>
> regards
> poonam
>
>
--
Nitin Pawar