Hadoop, mail # user - Re: formatting hdfs without user interaction

Re: formatting hdfs without user interaction

Arpit Gupta 2011-09-22, 21:46

you could try

echo yes | bin/hadoop namenode -format

--
Arpit
[EMAIL PROTECTED]
On Sep 22, 2011, at 2:43 PM, <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I am trying to automate formatting an HDFS volume. Is there any way to do this without the interaction (and using expect)?
>
> Cheers,
> Ivan

Re: formatting hdfs without user interaction

Raj V 2011-09-22, 21:52

Ian
"echo 'Y' | hadoop namenode -format"

should work.,

Raj
>________________________________
>From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Sent: Thursday, September 22, 2011 2:43 PM
>Subject: formatting hdfs without user interaction
>
>Hello,
>
>I am trying to automate formatting an HDFS volume. Is there any way to do this without the interaction (and using expect)?
>
>Cheers,
>Ivan
>
>
>

Re: formatting hdfs without user interaction

Ivan.Novick@... 2011-09-22, 21:59

Hi Raj and Arpit,

I think the fact that there are 2 confirmations is causing it not to work:

echo "Y" | hadoop namenode -format
11/09/22 14:57:10 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = hdm1/172.28.9.250
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.204.0
STARTUP_MSG: build = git://hrt8n35.cc1.ygridcore.net/ on branch
branch-0.20-security-204 -r 65e258bf0813ac2b15bb4c954660eaf9e8fba141;
compiled by 'hortonow' on Thu Aug 25 23:25:52 UTC 2011
************************************************************/
Re-format filesystem in /data/local/hdfs/name ? (Y or N) Re-format
filesystem in /data/remote/hdfs/name ? (Y or N) Format aborted in
/data/remote/hdfs/name
11/09/22 14:57:10 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hdm1/172.28.9.250
************************************************************/
Heap
par new generation total 2831168K, used 150996K [0x00002aaaae210000,
0x00002aab6e210000, 0x00002aab6e210000)
  eden space 2516608K, 6% used [0x00002aaaae210000, 0x00002aaab75852d0,
0x00002aab47bb0000)
  from space 314560K, 0% used [0x00002aab47bb0000, 0x00002aab47bb0000,
0x00002aab5aee0000)
  to space 314560K, 0% used [0x00002aab5aee0000, 0x00002aab5aee0000,
0x00002aab6e210000)
concurrent mark-sweep generation total 30408704K, used 0K
[0x00002aab6e210000, 0x00002ab2ae210000, 0x00002ab2ae210000)
concurrent-mark-sweep perm gen total 21248K, used 7333K
[0x00002ab2ae210000, 0x00002ab2af6d0000, 0x00002ab2b3610000)
Cheers,
Ivan

On 9/22/11 2:52 PM, "Raj V" <[EMAIL PROTECTED]> wrote:

>Ian
>"echo 'Y' | hadoop namenode -format"
>
>should work.,
>
>Raj
>
>
>>________________________________
>>From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>To: [EMAIL PROTECTED]
>>Sent: Thursday, September 22, 2011 2:43 PM
>>Subject: formatting hdfs without user interaction
>>
>>Hello,
>>
>>I am trying to automate formatting an HDFS volume. Is there any way to
>>do this without the interaction (and using expect)?
>>
>>Cheers,
>>Ivan
>>
>>

RE: formatting hdfs without user interaction

Adam Shook 2011-09-22, 22:02

yes | hadoop namenode -format

The "yes" program simply outputs 'y' a bunch of times. "echo yes" will just print "yes" to stdout.

-- Adam

-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 22, 2011 6:00 PM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re: formatting hdfs without user interaction

Hi Raj and Arpit,

I think the fact that there are 2 confirmations is causing it not to work:

echo "Y" | hadoop namenode -format
11/09/22 14:57:10 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = hdm1/172.28.9.250
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.204.0
STARTUP_MSG: build = git://hrt8n35.cc1.ygridcore.net/ on branch
branch-0.20-security-204 -r 65e258bf0813ac2b15bb4c954660eaf9e8fba141;
compiled by 'hortonow' on Thu Aug 25 23:25:52 UTC 2011
************************************************************/
Re-format filesystem in /data/local/hdfs/name ? (Y or N) Re-format
filesystem in /data/remote/hdfs/name ? (Y or N) Format aborted in
/data/remote/hdfs/name
11/09/22 14:57:10 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hdm1/172.28.9.250
************************************************************/
Heap
par new generation total 2831168K, used 150996K [0x00002aaaae210000,
0x00002aab6e210000, 0x00002aab6e210000)
  eden space 2516608K, 6% used [0x00002aaaae210000, 0x00002aaab75852d0,
0x00002aab47bb0000)
  from space 314560K, 0% used [0x00002aab47bb0000, 0x00002aab47bb0000,
0x00002aab5aee0000)
  to space 314560K, 0% used [0x00002aab5aee0000, 0x00002aab5aee0000,
0x00002aab6e210000)
concurrent mark-sweep generation total 30408704K, used 0K
[0x00002aab6e210000, 0x00002ab2ae210000, 0x00002ab2ae210000)
concurrent-mark-sweep perm gen total 21248K, used 7333K
[0x00002ab2ae210000, 0x00002ab2af6d0000, 0x00002ab2b3610000)
Cheers,
Ivan

On 9/22/11 2:52 PM, "Raj V" <[EMAIL PROTECTED]> wrote:

>Ian
>"echo 'Y' | hadoop namenode -format"
>
>should work.,
>
>Raj
>
>
>>________________________________
>>From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>To: [EMAIL PROTECTED]
>>Sent: Thursday, September 22, 2011 2:43 PM
>>Subject: formatting hdfs without user interaction
>>
>>Hello,
>>
>>I am trying to automate formatting an HDFS volume. Is there any way to
>>do this without the interaction (and using expect)?
>>
>>Cheers,
>>Ivan
>>
>>
-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1410 / Virus Database: 1520/3912 - Release Date: 09/22/11

Re: formatting hdfs without user interaction

Arpit Gupta 2011-09-22, 22:17

The reason you are getting multiple prompts is that you have multiple dir's defined in the dfs.name.dir.

A simple expect script would take care of this.

#!/usr/bin/expect -f
spawn /bin/hadoop namenode -format
expect "Re-format filesystem in"
send Y\n
expect "Re-format filesystem in"
send Y\n
interact
You might have to put the above in a loop based on how many dirs are defined in the dfs.name.dir

--
Arpit
[EMAIL PROTECTED]
On Sep 22, 2011, at 2:59 PM, <[EMAIL PROTECTED]> wrote:

> Hi Raj and Arpit,
>
> I think the fact that there are 2 confirmations is causing it not to work:
>
> echo "Y" | hadoop namenode -format
> 11/09/22 14:57:10 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = hdm1/172.28.9.250
> STARTUP_MSG: args = [-format]
> STARTUP_MSG: version = 0.20.204.0
> STARTUP_MSG: build = git://hrt8n35.cc1.ygridcore.net/ on branch
> branch-0.20-security-204 -r 65e258bf0813ac2b15bb4c954660eaf9e8fba141;
> compiled by 'hortonow' on Thu Aug 25 23:25:52 UTC 2011
> ************************************************************/
> Re-format filesystem in /data/local/hdfs/name ? (Y or N) Re-format
> filesystem in /data/remote/hdfs/name ? (Y or N) Format aborted in
> /data/remote/hdfs/name
> 11/09/22 14:57:10 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at hdm1/172.28.9.250
> ************************************************************/
> Heap
> par new generation total 2831168K, used 150996K [0x00002aaaae210000,
> 0x00002aab6e210000, 0x00002aab6e210000)
> eden space 2516608K, 6% used [0x00002aaaae210000, 0x00002aaab75852d0,
> 0x00002aab47bb0000)
> from space 314560K, 0% used [0x00002aab47bb0000, 0x00002aab47bb0000,
> 0x00002aab5aee0000)
> to space 314560K, 0% used [0x00002aab5aee0000, 0x00002aab5aee0000,
> 0x00002aab6e210000)
> concurrent mark-sweep generation total 30408704K, used 0K
> [0x00002aab6e210000, 0x00002ab2ae210000, 0x00002ab2ae210000)
> concurrent-mark-sweep perm gen total 21248K, used 7333K
> [0x00002ab2ae210000, 0x00002ab2af6d0000, 0x00002ab2b3610000)
>
>
> Cheers,
> Ivan
>
> On 9/22/11 2:52 PM, "Raj V" <[EMAIL PROTECTED]> wrote:
>
>> Ian
>> "echo 'Y' | hadoop namenode -format"
>>
>> should work.,
>>
>> Raj
>>
>>
>>> ________________________________
>>> From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>> To: [EMAIL PROTECTED]
>>> Sent: Thursday, September 22, 2011 2:43 PM
>>> Subject: formatting hdfs without user interaction
>>>
>>> Hello,
>>>
>>> I am trying to automate formatting an HDFS volume. Is there any way to
>>> do this without the interaction (and using expect)?
>>>
>>> Cheers,
>>> Ivan
>>>
>>>
>

Re: formatting hdfs without user interaction

Ivan.Novick@... 2011-09-22, 22:22

Yeah I have a secondary namenode as well so 2 directories.

I was trying to avoid expect if possible..... But this is always an option.

Cheers,
Ivan

On 9/22/11 3:17 PM, "Arpit Gupta" <[EMAIL PROTECTED]> wrote:

>The reason you are getting multiple prompts is that you have multiple
>dir's defined in the dfs.name.dir.
>
>A simple expect script would take care of this.
>
>#!/usr/bin/expect -f
>spawn /bin/hadoop namenode -format
>expect "Re-format filesystem in"
>send Y\n
>expect "Re-format filesystem in"
>send Y\n
>interact
>
>
>You might have to put the above in a loop based on how many dirs are
>defined in the dfs.name.dir
>
>--
>Arpit
>[EMAIL PROTECTED]
>
>
>
>
>On Sep 22, 2011, at 2:59 PM, <[EMAIL PROTECTED]> wrote:
>
>> Hi Raj and Arpit,
>>
>> I think the fact that there are 2 confirmations is causing it not to
>>work:
>>
>> echo "Y" | hadoop namenode -format
>> 11/09/22 14:57:10 INFO namenode.NameNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting NameNode
>> STARTUP_MSG: host = hdm1/172.28.9.250
>> STARTUP_MSG: args = [-format]
>> STARTUP_MSG: version = 0.20.204.0
>> STARTUP_MSG: build = git://hrt8n35.cc1.ygridcore.net/ on branch
>> branch-0.20-security-204 -r 65e258bf0813ac2b15bb4c954660eaf9e8fba141;
>> compiled by 'hortonow' on Thu Aug 25 23:25:52 UTC 2011
>> ************************************************************/
>> Re-format filesystem in /data/local/hdfs/name ? (Y or N) Re-format
>> filesystem in /data/remote/hdfs/name ? (Y or N) Format aborted in
>> /data/remote/hdfs/name
>> 11/09/22 14:57:10 INFO namenode.NameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down NameNode at hdm1/172.28.9.250
>> ************************************************************/
>> Heap
>> par new generation total 2831168K, used 150996K [0x00002aaaae210000,
>> 0x00002aab6e210000, 0x00002aab6e210000)
>> eden space 2516608K, 6% used [0x00002aaaae210000, 0x00002aaab75852d0,
>> 0x00002aab47bb0000)
>> from space 314560K, 0% used [0x00002aab47bb0000, 0x00002aab47bb0000,
>> 0x00002aab5aee0000)
>> to space 314560K, 0% used [0x00002aab5aee0000, 0x00002aab5aee0000,
>> 0x00002aab6e210000)
>> concurrent mark-sweep generation total 30408704K, used 0K
>> [0x00002aab6e210000, 0x00002ab2ae210000, 0x00002ab2ae210000)
>> concurrent-mark-sweep perm gen total 21248K, used 7333K
>> [0x00002ab2ae210000, 0x00002ab2af6d0000, 0x00002ab2b3610000)
>>
>>
>> Cheers,
>> Ivan
>>
>> On 9/22/11 2:52 PM, "Raj V" <[EMAIL PROTECTED]> wrote:
>>
>>> Ian
>>> "echo 'Y' | hadoop namenode -format"
>>>
>>> should work.,
>>>
>>> Raj
>>>
>>>
>>>> ________________________________
>>>> From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>>> To: [EMAIL PROTECTED]
>>>> Sent: Thursday, September 22, 2011 2:43 PM
>>>> Subject: formatting hdfs without user interaction
>>>>
>>>> Hello,
>>>>
>>>> I am trying to automate formatting an HDFS volume. Is there any way
>>>>to
>>>> do this without the interaction (and using expect)?
>>>>
>>>> Cheers,
>>>> Ivan
>>>>
>>>>
>>
>
>

Re: formatting hdfs without user interaction

Ivan.Novick@... 2011-09-22, 22:23

Hi Adam,

Well the "yes" program prints lower case y's and apparently only captial Y
is accepted.

But by creating my out "Yes" program that spews Y's to stdout it works :)

Cheers,
Ivan

On 9/22/11 3:02 PM, "Adam Shook" <[EMAIL PROTECTED]> wrote:

>yes | hadoop namenode -format
>
>The "yes" program simply outputs 'y' a bunch of times. "echo yes" will
>just print "yes" to stdout.
>
>-- Adam
>
>-----Original Message-----
>From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
>Sent: Thursday, September 22, 2011 6:00 PM
>To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
>Subject: Re: formatting hdfs without user interaction
>
>Hi Raj and Arpit,
>
>I think the fact that there are 2 confirmations is causing it not to work:
>
>echo "Y" | hadoop namenode -format
>11/09/22 14:57:10 INFO namenode.NameNode: STARTUP_MSG:
>/************************************************************
>STARTUP_MSG: Starting NameNode
>STARTUP_MSG: host = hdm1/172.28.9.250
>STARTUP_MSG: args = [-format]
>STARTUP_MSG: version = 0.20.204.0
>STARTUP_MSG: build = git://hrt8n35.cc1.ygridcore.net/ on branch
>branch-0.20-security-204 -r 65e258bf0813ac2b15bb4c954660eaf9e8fba141;
>compiled by 'hortonow' on Thu Aug 25 23:25:52 UTC 2011
>************************************************************/
>Re-format filesystem in /data/local/hdfs/name ? (Y or N) Re-format
>filesystem in /data/remote/hdfs/name ? (Y or N) Format aborted in
>/data/remote/hdfs/name
>11/09/22 14:57:10 INFO namenode.NameNode: SHUTDOWN_MSG:
>/************************************************************
>SHUTDOWN_MSG: Shutting down NameNode at hdm1/172.28.9.250
>************************************************************/
>Heap
> par new generation total 2831168K, used 150996K [0x00002aaaae210000,
>0x00002aab6e210000, 0x00002aab6e210000)
> eden space 2516608K, 6% used [0x00002aaaae210000, 0x00002aaab75852d0,
>0x00002aab47bb0000)
> from space 314560K, 0% used [0x00002aab47bb0000, 0x00002aab47bb0000,
>0x00002aab5aee0000)
> to space 314560K, 0% used [0x00002aab5aee0000, 0x00002aab5aee0000,
>0x00002aab6e210000)
> concurrent mark-sweep generation total 30408704K, used 0K
>[0x00002aab6e210000, 0x00002ab2ae210000, 0x00002ab2ae210000)
> concurrent-mark-sweep perm gen total 21248K, used 7333K
>[0x00002ab2ae210000, 0x00002ab2af6d0000, 0x00002ab2b3610000)
>
>
>Cheers,
>Ivan
>
>On 9/22/11 2:52 PM, "Raj V" <[EMAIL PROTECTED]> wrote:
>
>>Ian
>>"echo 'Y' | hadoop namenode -format"
>>
>>should work.,
>>
>>Raj
>>
>>
>>>________________________________
>>>From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>>To: [EMAIL PROTECTED]
>>>Sent: Thursday, September 22, 2011 2:43 PM
>>>Subject: formatting hdfs without user interaction
>>>
>>>Hello,
>>>
>>>I am trying to automate formatting an HDFS volume. Is there any way to
>>>do this without the interaction (and using expect)?
>>>
>>>Cheers,
>>>Ivan
>>>
>>>
>
>
>-----
>No virus found in this message.
>Checked by AVG - www.avg.com
>Version: 10.0.1410 / Virus Database: 1520/3912 - Release Date: 09/22/11
>

Re: formatting hdfs without user interaction

Harsh J 2011-09-23, 03:48

Ivan,

Writing your own program was overkill.

The 'yes' coreutil is pretty silly, but nifty at the same time. It
accepts an argument, which it would repeat infinitely.

So:

$ yes Y | hadoop namenode -format

Would do it for you.

(Note that in the future release, saner answers will be acceptable,
i.e. y instead of strictly Y, etc.)

Also, two other things:

- What do you mean by 'Yeah I have a secondary namenode as well so 2
directories.'? A secondary namenode uses different directories than
dfs.name.dir.
- The prompt only appears when it detects a 'reformat' being happening
- which is very dangerous to do non-interactively. If you do the
-format the first time, on clean dfs.name.dir setups, you will never
receive a prompt.

On Fri, Sep 23, 2011 at 3:53 AM, <[EMAIL PROTECTED]> wrote:
> Hi Adam,
>
> Well the "yes" program prints lower case y's and apparently only captial Y
> is accepted.
>
> But by creating my out "Yes" program that spews Y's to stdout it works :)
>
> Cheers,
> Ivan
>
> On 9/22/11 3:02 PM, "Adam Shook" <[EMAIL PROTECTED]> wrote:
>
>>yes | hadoop namenode -format
>>
>>The "yes" program simply outputs 'y' a bunch of times. "echo yes" will
>>just print "yes" to stdout.
>>
>>-- Adam
>>
>>-----Original Message-----
>>From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
>>Sent: Thursday, September 22, 2011 6:00 PM
>>To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
>>Subject: Re: formatting hdfs without user interaction
>>
>>Hi Raj and Arpit,
>>
>>I think the fact that there are 2 confirmations is causing it not to work:
>>
>>echo "Y" | hadoop namenode -format
>>11/09/22 14:57:10 INFO namenode.NameNode: STARTUP_MSG:
>>/************************************************************
>>STARTUP_MSG: Starting NameNode
>>STARTUP_MSG: host = hdm1/172.28.9.250
>>STARTUP_MSG: args = [-format]
>>STARTUP_MSG: version = 0.20.204.0
>>STARTUP_MSG: build = git://hrt8n35.cc1.ygridcore.net/ on branch
>>branch-0.20-security-204 -r 65e258bf0813ac2b15bb4c954660eaf9e8fba141;
>>compiled by 'hortonow' on Thu Aug 25 23:25:52 UTC 2011
>>************************************************************/
>>Re-format filesystem in /data/local/hdfs/name ? (Y or N) Re-format
>>filesystem in /data/remote/hdfs/name ? (Y or N) Format aborted in
>>/data/remote/hdfs/name
>>11/09/22 14:57:10 INFO namenode.NameNode: SHUTDOWN_MSG:
>>/************************************************************
>>SHUTDOWN_MSG: Shutting down NameNode at hdm1/172.28.9.250
>>************************************************************/
>>Heap
>> par new generation total 2831168K, used 150996K [0x00002aaaae210000,
>>0x00002aab6e210000, 0x00002aab6e210000)
>> eden space 2516608K, 6% used [0x00002aaaae210000, 0x00002aaab75852d0,
>>0x00002aab47bb0000)
>> from space 314560K, 0% used [0x00002aab47bb0000, 0x00002aab47bb0000,
>>0x00002aab5aee0000)
>> to space 314560K, 0% used [0x00002aab5aee0000, 0x00002aab5aee0000,
>>0x00002aab6e210000)
>> concurrent mark-sweep generation total 30408704K, used 0K
>>[0x00002aab6e210000, 0x00002ab2ae210000, 0x00002ab2ae210000)
>> concurrent-mark-sweep perm gen total 21248K, used 7333K
>>[0x00002ab2ae210000, 0x00002ab2af6d0000, 0x00002ab2b3610000)
>>
>>
>>Cheers,
>>Ivan
>>
>>On 9/22/11 2:52 PM, "Raj V" <[EMAIL PROTECTED]> wrote:
>>
>>>Ian
>>>"echo 'Y' | hadoop namenode -format"
>>>
>>>should work.,
>>>
>>>Raj
>>>
>>>
>>>>________________________________
>>>>From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>>>To: [EMAIL PROTECTED]
>>>>Sent: Thursday, September 22, 2011 2:43 PM
>>>>Subject: formatting hdfs without user interaction
>>>>
>>>>Hello,
>>>>
>>>>I am trying to automate formatting an HDFS volume. Is there any way to
>>>>do this without the interaction (and using expect)?
>>>>
>>>>Cheers,
>>>>Ivan
>>>>
>>>>
>>
>>
>>-----
>>No virus found in this message.
>>Checked by AVG - www.avg.com
>>Version: 10.0.1410 / Virus Database: 1520/3912 - Release Date: 09/22/11
>>
>
>

--
Harsh J

Re: formatting hdfs without user interaction

Ivan.Novick@... 2011-09-23, 15:52

Hi Harsh,

On 9/22/11 8:48 PM, "Harsh J" <[EMAIL PROTECTED]> wrote:

>Ivan,
>
>Writing your own program was overkill.
>
>The 'yes' coreutil is pretty silly, but nifty at the same time. It
>accepts an argument, which it would repeat infinitely.
>
>So:
>
>$ yes Y | hadoop namenode -format
>
>Would do it for you.

Nice! I read the man page for yes too quickly and did not see that
option. Thanks!
>(Note that in the future release, saner answers will be acceptable,
>i.e. y instead of strictly Y, etc.)

Y/y/yes/YES would all seem like good things to accept :)
>Also, two other things:
>
>- What do you mean by 'Yeah I have a secondary namenode as well so 2
>directories.'? A secondary namenode uses different directories than
>dfs.name.dir.

Which parameter are you referring to? I am planning on using 2 directories
in dfs.name.dir, one is local and the other is an NFS mount of a 2nd
machine running the secondary namenode.
>- The prompt only appears when it detects a 'reformat' being happening
>- which is very dangerous to do non-interactively. If you do the
>-format the first time, on clean dfs.name.dir setups, you will never
>receive a prompt.

Yeah I am creating some automation, so it needs to be able to wipe out an
existing filesystem and start over......
Cheers,
Ivan

>

Re: formatting hdfs without user interaction

Edward Capriolo 2011-09-23, 16:01

On Fri, Sep 23, 2011 at 11:52 AM, <[EMAIL PROTECTED]> wrote:

> Hi Harsh,
>
> On 9/22/11 8:48 PM, "Harsh J" <[EMAIL PROTECTED]> wrote:
>
> >Ivan,
> >
> >Writing your own program was overkill.
> >
> >The 'yes' coreutil is pretty silly, but nifty at the same time. It
> >accepts an argument, which it would repeat infinitely.
> >
> >So:
> >
> >$ yes Y | hadoop namenode -format
> >
> >Would do it for you.
>
> Nice! I read the man page for yes too quickly and did not see that
> option. Thanks!
>
>
> >(Note that in the future release, saner answers will be acceptable,
> >i.e. y instead of strictly Y, etc.)
>
> Y/y/yes/YES would all seem like good things to accept :)
>
>
> >Also, two other things:
> >
> >- What do you mean by 'Yeah I have a secondary namenode as well so 2
> >directories.'? A secondary namenode uses different directories than
> >dfs.name.dir.
>
> Which parameter are you referring to? I am planning on using 2 directories
> in dfs.name.dir, one is local and the other is an NFS mount of a 2nd
> machine running the secondary namenode.
>
>
> >- The prompt only appears when it detects a 'reformat' being happening
> >- which is very dangerous to do non-interactively. If you do the
> >-format the first time, on clean dfs.name.dir setups, you will never
> >receive a prompt.
>
> Yeah I am creating some automation, so it needs to be able to wipe out an
> existing filesystem and start over......
>
>
> Cheers,
> Ivan
>
>
>
> >
>
>
You might want to try expect scripting. You open a stream to the process and
then can wait for prompts and print replies. expect also has this feature
autoexpect which is a shell that records the streams and turns your teminal
interaction into an expect script.

Re: formatting hdfs without user interaction

Ivan.Novick@... 2011-09-23, 16:15

On 9/23/11 9:01 AM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote:

>On Fri, Sep 23, 2011 at 11:52 AM, <[EMAIL PROTECTED]> wrote:
>
>> Hi Harsh,
>>
>> On 9/22/11 8:48 PM, "Harsh J" <[EMAIL PROTECTED]> wrote:
>>
>> >Ivan,
>> >
>> >Writing your own program was overkill.
>> >
>> >The 'yes' coreutil is pretty silly, but nifty at the same time. It
>> >accepts an argument, which it would repeat infinitely.
>> >
>> >So:
>> >
>> >$ yes Y | hadoop namenode -format
>> >
>> >Would do it for you.
>>
>> Nice! I read the man page for yes too quickly and did not see that
>> option. Thanks!
>>
>>...... SNIP ...............
>>
>You might want to try expect scripting. You open a stream to the process
>and
>then can wait for prompts and print replies. expect also has this feature
>autoexpect which is a shell that records the streams and turns your
>teminal
>interaction into an expect script.

Yeah, I have used expect via a python library numerous times, but expect
is a bit brittle and error prone, so I try to avoid it whenever possible.

Cheers,
Ivan

Re: formatting hdfs without user interaction

Harsh J 2011-09-23, 16:46

Ivan,

On Fri, Sep 23, 2011 at 9:22 PM, <[EMAIL PROTECTED]> wrote:
[snip]
> Which parameter are you referring to? I am planning on using 2 directories
> in dfs.name.dir, one is local and the other is an NFS mount of a 2nd
> machine running the secondary namenode.

I'm slightly confused here. Do you mean to say a 'SecondaryNameNode',
which is a form of checkpointing daemon available as part of Hadoop,
or do you mean to say, loosely, 'secondary namenode', which would
imply a machine that can step in as a NameNode once you start it (in
case the primary fails)?

For the former, the properties are the ones that begin with
"fs.checkpoint.*" in 0.20.x versions, which you can checkout.

--
Harsh J

Re: formatting hdfs without user interaction

Ivan.Novick@... 2011-09-23, 17:25

On 9/23/11 9:46 AM, "Harsh J" <[EMAIL PROTECTED]> wrote:

>Ivan,
>
>On Fri, Sep 23, 2011 at 9:22 PM, <[EMAIL PROTECTED]> wrote:
>[snip]
>> Which parameter are you referring to? I am planning on using 2
>>directories
>> in dfs.name.dir, one is local and the other is an NFS mount of a 2nd
>> machine running the secondary namenode.
>
>I'm slightly confused here. Do you mean to say a 'SecondaryNameNode',
>which is a form of checkpointing daemon available as part of Hadoop,
>or do you mean to say, loosely, 'secondary namenode', which would
>imply a machine that can step in as a NameNode once you start it (in
>case the primary fails)?
>
>For the former, the properties are the ones that begin with
>"fs.checkpoint.*" in 0.20.x versions, which you can checkout.

I mean both... A 2nd node which can step in as a name node and also that
node will run the checkpointing daemon. I have 2 directories in
dfs.name.dir one is local and one is remote on the 2nd machine so we have
a copy of the data if we lose the first machine.

Cheers,
Ivan