Hello Ted,

Thanks yours below. I do run hadoop on EC2 as well. And I use internal host-addresses ($hostname -f) for configuration of core-site/hdfs-site/slaves/master files.

The only time I switch to external address i.e. ec2-......  is when I want to access my nodes vie the web,i.e. external address  - ec2......:50070 or ec2......:50030

Rgds,
AK47
yinghua,

One other thing to note here is that on EC2 computer are given two hostnames, one internal and one external, you need to use the external.  These host names are listed in the ec2 configuration pane on the AWS management console.
Ted Reynolds
Technical Support Engineer
Hortonworks
Work Phone: 408-645-7079

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel

Hi, Ted and Andy

I tried both internal and external hostnames. They both worked. But I will
elect external host name since this way I can browsing the hdfs file
structure and other information from a computer not in the cluster.

Thanks!

On Fri, Nov 9, 2012 at 1:03 PM, Kartashov, Andy <[EMAIL PROTECTED]>wrote:

>  Hello Ted,
>
>
>
> Thanks yours below. I do run hadoop on EC2 as well. And I use internal
> host-addresses ($hostname –f) for configuration of
> core-site/hdfs-site/slaves/master files.
>
>
>
> The only time I switch to external address i.e. ec2-……  is when I want to
> access my nodes vie the web,i.e. external address  - ec2……:50070 or
> ec2……:50030
>
>
>
> Rgds,
>
> AK47
>
>
>
>
>
> yinghua,
>
>
>
> One other thing to note here is that on EC2 computer are given two
> hostnames, one internal and one external, you need to use the external.
>  These host names are listed in the ec2 configuration pane on the AWS
> management console.
>
>
>
>
>  Ted Reynolds
>
> Technical Support Engineer
>
> Hortonworks
>
> Work Phone: 408-645-7079
>
>
>   NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>

--
Regards,

Yinghua

On 10 November 2012 00:48, yinghua hu <[EMAIL PROTECTED]> wrote:

> Hi, Ted and Andy
>
> I tried both internal and external hostnames. They both worked. But I will
> elect external host name since this way I can browsing the hdfs file
> structure and other information from a computer not in the cluster.
>
> Thanks!
>

I wouldn't recommend this:

If you hook up your DNs to the NN via the external addresses, Amazon will
bill you for external traffic, which could be an unpleasant surprise at the
end of the month.

If you deploy using Apache Whirr then it gives you a command to run an SSH
proxy.

I've filed a JIRA for the webhdfs feature in Hadoop 1.0.3+ -which offers a
way to get at the filesystem via HTTP to not have the NN issue 302
redirects to the specific DN, but instead act as a proxy; there is a
cluster proxy service (with the same API) that can do this, but for small
EC2 clusters using the NN directly would be easier. What I haven't done is
implemented it -if anyone else wants to I'll gladly review the code (and
its tests)
-Steve

Thanks Steve,

Your suggestion is very helpful. But I only have terminal access to the EC2
NN. To run internet browser there will be a lot of more work.

Since my cluster will be shut down at night and weekend. I will prefer to
have an IP that does not change. How do you think of using EC2 Elastic IP?

On Sat, Nov 10, 2012 at 2:38 AM, Steve Loughran <[EMAIL PROTECTED]>wrote:

>
>
> On 10 November 2012 00:48, yinghua hu <[EMAIL PROTECTED]> wrote:
>
>> Hi, Ted and Andy
>>
>> I tried both internal and external hostnames. They both worked. But I
>> will elect external host name since this way I can browsing the hdfs file
>> structure and other information from a computer not in the cluster.
>>
>> Thanks!
>>
>
> I wouldn't recommend this:
>
> If you hook up your DNs to the NN via the external addresses, Amazon will
> bill you for external traffic, which could be an unpleasant surprise at the
> end of the month.
>
> If you deploy using Apache Whirr then it gives you a command to run an SSH
> proxy.
>
> I've filed a JIRA for the webhdfs feature in Hadoop 1.0.3+ -which offers a
> way to get at the filesystem via HTTP to not have the NN issue 302
> redirects to the specific DN, but instead act as a proxy; there is a
> cluster proxy service (with the same API) that can do this, but for small
> EC2 clusters using the NN directly would be easier. What I haven't done is
> implemented it -if anyone else wants to I'll gladly review the code (and
> its tests)
>
>
> -Steve
>
>
--
Regards,

Yinghua

On 10 November 2012 18:30, yinghua hu <[EMAIL PROTECTED]> wrote:

> Thanks Steve,
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>

the SSH proxy sets up something that appears to be a SOCKs proxy -but which
routes to the far end; you set your browser to go through this proxy
(foxyproxy is a good firefox plugin here), and then your browser has access
to all the machines

see http://www.linuxjournal.com/content/use-ssh-create-http-proxy

>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
not tried that; I have used dyndns.org to self register an IP address that
changes every time you bring up an instance, but whose browser bookmarks
can remain fixed

Hi, Steve

These are very helpful suggestions! Thanks!

I will try using SSH proxy and see how it goes.

On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <[EMAIL PROTECTED]>wrote:

>
>
> On 10 November 2012 18:30, yinghua hu <[EMAIL PROTECTED]> wrote:
>
>> Thanks Steve,
>>
>> Your suggestion is very helpful. But I only have terminal access to the
>> EC2 NN. To run internet browser there will be a lot of more work.
>>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>>
>> Since my cluster will be shut down at night and weekend. I will prefer to
>> have an IP that does not change. How do you think of using EC2 Elastic IP?
>>
>>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>

--
Regards,

Yinghua

Good to know it works as I well. Thanks for sharing.
From: yinghua hu [mailto:[EMAIL PROTECTED]]
Sent: Friday, November 09, 2012 7:48 PM
To: [EMAIL PROTECTED]
Subject: Re: error running pi program

Hi, Ted and Andy

I tried both internal and external hostnames. They both worked. But I will elect external host name since this way I can browsing the hdfs file structure and other information from a computer not in the cluster.

Thanks!
On Fri, Nov 9, 2012 at 1:03 PM, Kartashov, Andy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hello Ted,

Thanks yours below. I do run hadoop on EC2 as well. And I use internal host-addresses ($hostname -f) for configuration of core-site/hdfs-site/slaves/master files.

The only time I switch to external address i.e. ec2-......  is when I want to access my nodes vie the web,i.e. external address  - ec2......:50070 or ec2......:50030

Rgds,
AK47
yinghua,

One other thing to note here is that on EC2 computer are given two hostnames, one internal and one external, you need to use the external.  These host names are listed in the ec2 configuration pane on the AWS management console.
Ted Reynolds
Technical Support Engineer
Hortonworks
Work Phone: 408-645-7079<tel:408-645-7079>

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

--
Regards,

Yinghua
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel