Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> HWI use on AWS/EMR


Copy link to this message
-
RE: HWI use on AWS/EMR
Hi Nitin (and others who have contributed)

Thanks again for the suggestions - however my original objective was to investigate HWI as an alternative to hive in CLI mode for less techie analysts to get their hands on data in S3. At the moment it's easier for them to start an EMR instance running hive than it is to spawn instances from other instances with the appropriate security group; so I'm letting them carry on with the former for now.

Tony
From: Nitin Pawar [mailto:[EMAIL PROTECTED]]
Sent: 21 January 2013 13:17
To: [EMAIL PROTECTED]
Subject: Re: HWI use on AWS/EMR

if you are not inside VPC ec2 does not let you change security groups.

so best option would be create a new ami from ur instance and then launch a new instance with a security group you newly created.
or change the default security group to have the changes you want

On Mon, Jan 21, 2013 at 6:43 PM, Tony Burton <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Thanks for the suggestion Nitin.

I can see the "Change Security Groups" option for my instance (when looking at the EC2 console), but it's greyed out. The only option I have on the EMR console for the instance is "Terminate Job".

Is your "Change Security Groups" option enabled? If so, any idea what steps you took in creating the instance to enable it?

From: Nitin Pawar [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: 21 January 2013 11:57

To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: HWI use on AWS/EMR

right click on the instance in your aws console and you  should be able to change the security group for the node.

This feature was not there previously, came in few months back

On Mon, Jan 21, 2013 at 5:09 PM, Tony Burton <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
I've tracked down this document on AWS:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html

which shows how to create security groups to associate with your AWS instances. Simple enough to create a rule - but is there anyway to associate a rule with an instance either when the instance is running, or at creation time using the AWS Management Console? Or can it only be done using the commandline client or by an http request?

Thanks,

Tony
From: Ariel Marcus [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: 18 January 2013 17:25

To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: HWI use on AWS/EMR

Hey Tony,

Nitin's approach would work for you as well. Amazon has very good documentation of their services. A quick google search should get you the right info.

Best,
Ariel

---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>

On Fri, Jan 18, 2013 at 12:17 PM, Tony Burton <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
If you could provide the steps to do this I'd be grateful - thanks.

Thanks for all your input so far, time to head home for the weekend so I'll try it again with a fresh EMR instance next week.

Tony

From: Nitin Pawar [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: 18 January 2013 17:08

To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: HWI use on AWS/EMR

ok so much discussion around this but why dont you open access to port 9999 for your specific machine (home/office) and access it normally without setting up port forwarding.

AWS does give you ip level access control in the security groups

On Fri, Jan 18, 2013 at 10:35 PM, Ariel Marcus <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
This should be localhost instead of proxy:

- In Connection/SSH/Tunnels I've added 9999 in the Source Port box, localhost:9999 in the Destination box. Not sure if this is correct.

Best,
Aril

Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>

On Fri, Jan 18, 2013 at 12:00 PM, Tony Burton <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:

Ok, thanks Ariel (and Dean in the next email!)

I've set up PuTTY (v0.62) as follows:

- EMR instance hostname in the Session/Host Name box (port left at 22, ok?)

- In Connection/SSH/Auth I've added my private key file, downloaded from AWS ages for this account.

- In Connection/SSH/Tunnels I've added 9999 in the Source Port box, proxy:9999 in the Destination box. Not sure if this is correct.

Hit the Open button, I can connect. (And I can see my changes in hive-default.xml, so I know I'm in the right place:)

Then pointing my browser at http://localhost:9999 I get the same timeout again. Have I set up everything right?

Also, following good advice I copied the config changes to hive-site.xml. How do I stop hive so I can restart with 'hive -service hwi' so that it picks up the new config?

Tony

From: Ariel Marcus [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: 18 January 2013 16:31

To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: HWI use on AWS/EMR

Hey Tony,

Port forwarding is the same as what Dean referred to as ssh tunneling.

Here is a website explaining how to set up port forwarding in putty (use port 9999):

http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html

You can edit either hive-default or hive-site but the general practice is to store changes from the default configuration in hive-site.

Best,

Ariel

Ariel Marcus, Consultant

www.openbi.com<http://www.openbi.com/> | [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>

150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>

On Fri, Jan 18, 2013 at