Hello Gurus,

I am managing a Hadoop Cluster to run some experiments.

The issue I am continuously facing is that the Master Node runs out of
disk space due to logs and data files.

I can monitor and delete log files. However, I cannot delete the HDFS data.

Thus, is there a way to force Hadoop not to save any HDFS data in the
Master Node?

Then I can use my master to handle the metadata only and store the logs.

Thanks & regards
Arko

Hello Arko,

          Add the dfs.data.dir property in your hdfs-site.xml file and
point it to
some other location.

For logs, do the same thing by modifying the following line in
hadoop-env.sh file :

# Where log files are stored.  $HADOOP_HOME/logs by default.
export HADOOP_LOG_DIR=/hadoop/hdfs/logs

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 1:02 AM, Arko Provo Mukherjee <
[EMAIL PROTECTED]> wrote:

> Hello Gurus,
>
> I am managing a Hadoop Cluster to run some experiments.
>
> The issue I am continuously facing is that the Master Node runs out of
> disk space due to logs and data files.
>
> I can monitor and delete log files. However, I cannot delete the HDFS data.
>
> Thus, is there a way to force Hadoop not to save any HDFS data in the
> Master Node?
>
> Then I can use my master to handle the metadata only and store the logs.
>
> Thanks & regards
> Arko
>

Hi Arko. Sounds like you may be running a DataNode on the NameNode which is
not recommended practice. Normally, the only files the NN stores are the
image and edits files. It does not store any actual HDFS data. If you must
run a DN on the NN, try turning down the logging in /conf/log4j.properties:

#hadoop.root.logger=INFO,console
#hadoop.root.logger=WARN,console
hadoop.root.logger=ERROR,console

Depending on the logging information you require, of course.

-Chuck
-----Original Message-----
From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, February 13, 2013 11:32 AM
To: [EMAIL PROTECTED]
Subject: Managing space in Master Node

Hello Gurus,

I am managing a Hadoop Cluster to run some experiments.

The issue I am continuously facing is that the Master Node runs out of disk
space due to logs and data files.

I can monitor and delete log files. However, I cannot delete the HDFS data.

Thus, is there a way to force Hadoop not to save any HDFS data in the Master
Node?

Then I can use my master to handle the metadata only and store the logs.

Thanks & regards
Arko
SDL Enterprise Technologies, Inc. - all rights reserved.  The information contained in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not the intended recipient of this mail, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please reply to the sender and destroy all copies of the message.
Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA

Hi,

Yeah, my NameNode is also seconding as a DataNode.

I would like to "turn off" this feature.

Request help regarding the same.

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]> wrote:
> Hi Arko. Sounds like you may be running a DataNode on the NameNode which is
> not recommended practice. Normally, the only files the NN stores are the
> image and edits files. It does not store any actual HDFS data. If you must
> run a DN on the NN, try turning down the logging in /conf/log4j.properties:
>
> #hadoop.root.logger=INFO,console
> #hadoop.root.logger=WARN,console
> hadoop.root.logger=ERROR,console
>
> Depending on the logging information you require, of course.
>
> -Chuck
>
>
> -----Original Message-----
> From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, February 13, 2013 11:32 AM
> To: [EMAIL PROTECTED]
> Subject: Managing space in Master Node
>
> Hello Gurus,
>
> I am managing a Hadoop Cluster to run some experiments.
>
> The issue I am continuously facing is that the Master Node runs out of disk
> space due to logs and data files.
>
> I can monitor and delete log files. However, I cannot delete the HDFS data.
>
> Thus, is there a way to force Hadoop not to save any HDFS data in the Master
> Node?
>
> Then I can use my master to handle the metadata only and store the logs.
>
> Thanks & regards
> Arko
> SDL Enterprise Technologies, Inc. - all rights reserved.  The information contained in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not the intended recipient of this mail, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please reply to the sender and destroy all copies of the message.
> Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA
>

You can specify the logging level as specified by Charles. But turning logs
off is never a good idea. Logs are really helpful in problem diagnosis,
which are eventual.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> Yeah, my NameNode is also seconding as a DataNode.
>
> I would like to "turn off" this feature.
>
> Request help regarding the same.
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]> wrote:
> > Hi Arko. Sounds like you may be running a DataNode on the NameNode which
> is
> > not recommended practice. Normally, the only files the NN stores are the
> > image and edits files. It does not store any actual HDFS data. If you
> must
> > run a DN on the NN, try turning down the logging in
> /conf/log4j.properties:
> >
> > #hadoop.root.logger=INFO,console
> > #hadoop.root.logger=WARN,console
> > hadoop.root.logger=ERROR,console
> >
> > Depending on the logging information you require, of course.
> >
> > -Chuck
> >
> >
> > -----Original Message-----
> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
> > Sent: Wednesday, February 13, 2013 11:32 AM
> > To: [EMAIL PROTECTED]
> > Subject: Managing space in Master Node
> >
> > Hello Gurus,
> >
> > I am managing a Hadoop Cluster to run some experiments.
> >
> > The issue I am continuously facing is that the Master Node runs out of
> disk
> > space due to logs and data files.
> >
> > I can monitor and delete log files. However, I cannot delete the HDFS
> data.
> >
> > Thus, is there a way to force Hadoop not to save any HDFS data in the
> Master
> > Node?
> >
> > Then I can use my master to handle the metadata only and store the logs.
> >
> > Thanks & regards
> > Arko
> > SDL Enterprise Technologies, Inc. - all rights reserved.  The
> information contained in this email may be confidential and/or legally
> privileged. It has been sent for the sole use of the intended recipient(s).
> If you are not the intended recipient of this mail, you are hereby notified
> that any unauthorized review, use, disclosure, dissemination, distribution,
> or copying of this communication, or any of its contents, is strictly
> prohibited. If you have received this communication in error, please reply
> to the sender and destroy all copies of the message.
> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880,
> USA
> >
>

Hi,

Thanks for the help!

However, I am still unsure about how to "turn off" the datanode
feature on the NN.

I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
a directory.

Should I just comment out that property? What would happen to the
current data that there is in the Master? Will it get removed
automatically?

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> You can specify the logging level as specified by Charles. But turning logs
> off is never a good idea. Logs are really helpful in problem diagnosis,
> which are eventual.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> Yeah, my NameNode is also seconding as a DataNode.
>>
>> I would like to "turn off" this feature.
>>
>> Request help regarding the same.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]> wrote:
>> > Hi Arko. Sounds like you may be running a DataNode on the NameNode which
>> > is
>> > not recommended practice. Normally, the only files the NN stores are the
>> > image and edits files. It does not store any actual HDFS data. If you
>> > must
>> > run a DN on the NN, try turning down the logging in
>> > /conf/log4j.properties:
>> >
>> > #hadoop.root.logger=INFO,console
>> > #hadoop.root.logger=WARN,console
>> > hadoop.root.logger=ERROR,console
>> >
>> > Depending on the logging information you require, of course.
>> >
>> > -Chuck
>> >
>> >
>> > -----Original Message-----
>> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
>> > Sent: Wednesday, February 13, 2013 11:32 AM
>> > To: [EMAIL PROTECTED]
>> > Subject: Managing space in Master Node
>> >
>> > Hello Gurus,
>> >
>> > I am managing a Hadoop Cluster to run some experiments.
>> >
>> > The issue I am continuously facing is that the Master Node runs out of
>> > disk
>> > space due to logs and data files.
>> >
>> > I can monitor and delete log files. However, I cannot delete the HDFS
>> > data.
>> >
>> > Thus, is there a way to force Hadoop not to save any HDFS data in the
>> > Master
>> > Node?
>> >
>> > Then I can use my master to handle the metadata only and store the logs.
>> >
>> > Thanks & regards
>> > Arko
>> > SDL Enterprise Technologies, Inc. - all rights reserved.  The
>> > information contained in this email may be confidential and/or legally
>> > privileged. It has been sent for the sole use of the intended recipient(s).
>> > If you are not the intended recipient of this mail, you are hereby notified
>> > that any unauthorized review, use, disclosure, dissemination, distribution,
>> > or copying of this communication, or any of its contents, is strictly
>> > prohibited. If you have received this communication in error, please reply
>> > to the sender and destroy all copies of the message.
>> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880,
>> > USA
>> >
>
>

Just don't run the DN daemon on that machine. Dedicate that machine to NN.
Remove the hostname of the NN machine from the 'slaves' file.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> Thanks for the help!
>
> However, I am still unsure about how to "turn off" the datanode
> feature on the NN.
>
> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
> a directory.
>
> Should I just comment out that property? What would happen to the
> current data that there is in the Master? Will it get removed
> automatically?
>
> Thanks & regards
> Arko
>
>
>
> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> > You can specify the logging level as specified by Charles. But turning
> logs
> > off is never a good idea. Logs are really helpful in problem diagnosis,
> > which are eventual.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> > <[EMAIL PROTECTED]> wrote:
> >>
> >> Hi,
> >>
> >> Yeah, my NameNode is also seconding as a DataNode.
> >>
> >> I would like to "turn off" this feature.
> >>
> >> Request help regarding the same.
> >>
> >> Thanks & regards
> >> Arko
> >>
> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]> wrote:
> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
> which
> >> > is
> >> > not recommended practice. Normally, the only files the NN stores are
> the
> >> > image and edits files. It does not store any actual HDFS data. If you
> >> > must
> >> > run a DN on the NN, try turning down the logging in
> >> > /conf/log4j.properties:
> >> >
> >> > #hadoop.root.logger=INFO,console
> >> > #hadoop.root.logger=WARN,console
> >> > hadoop.root.logger=ERROR,console
> >> >
> >> > Depending on the logging information you require, of course.
> >> >
> >> > -Chuck
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
> >> > Sent: Wednesday, February 13, 2013 11:32 AM
> >> > To: [EMAIL PROTECTED]
> >> > Subject: Managing space in Master Node
> >> >
> >> > Hello Gurus,
> >> >
> >> > I am managing a Hadoop Cluster to run some experiments.
> >> >
> >> > The issue I am continuously facing is that the Master Node runs out of
> >> > disk
> >> > space due to logs and data files.
> >> >
> >> > I can monitor and delete log files. However, I cannot delete the HDFS
> >> > data.
> >> >
> >> > Thus, is there a way to force Hadoop not to save any HDFS data in the
> >> > Master
> >> > Node?
> >> >
> >> > Then I can use my master to handle the metadata only and store the
> logs.
> >> >
> >> > Thanks & regards
> >> > Arko
> >> > SDL Enterprise Technologies, Inc. - all rights reserved.  The
> >> > information contained in this email may be confidential and/or legally
> >> > privileged. It has been sent for the sole use of the intended
> recipient(s).
> >> > If you are not the intended recipient of this mail, you are hereby
> notified
> >> > that any unauthorized review, use, disclosure, dissemination,
> distribution,
> >> > or copying of this communication, or any of its contents, is strictly
> >> > prohibited. If you have received this communication in error, please
> reply
> >> > to the sender and destroy all copies of the message.
> >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA
> 01880,
> >> > USA
> >> >
> >
> >
>

Thanks!
However, in that case, how do I handle the current data that is already in NN?
Do I have to reformat HDFS?
Thanks & regards
Arko

On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Just don't run the DN daemon on that machine. Dedicate that machine to NN.
> Remove the hostname of the NN machine from the 'slaves' file.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> Thanks for the help!
>>
>> However, I am still unsure about how to "turn off" the datanode
>> feature on the NN.
>>
>> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>> a directory.
>>
>> Should I just comment out that property? What would happen to the
>> current data that there is in the Master? Will it get removed
>> automatically?
>>
>> Thanks & regards
>> Arko
>>
>>
>>
>> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> wrote:
>> > You can specify the logging level as specified by Charles. But turning
>> > logs
>> > off is never a good idea. Logs are really helpful in problem diagnosis,
>> > which are eventual.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> > <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Yeah, my NameNode is also seconding as a DataNode.
>> >>
>> >> I would like to "turn off" this feature.
>> >>
>> >> Request help regarding the same.
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]> wrote:
>> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
>> >> > which
>> >> > is
>> >> > not recommended practice. Normally, the only files the NN stores are
>> >> > the
>> >> > image and edits files. It does not store any actual HDFS data. If you
>> >> > must
>> >> > run a DN on the NN, try turning down the logging in
>> >> > /conf/log4j.properties:
>> >> >
>> >> > #hadoop.root.logger=INFO,console
>> >> > #hadoop.root.logger=WARN,console
>> >> > hadoop.root.logger=ERROR,console
>> >> >
>> >> > Depending on the logging information you require, of course.
>> >> >
>> >> > -Chuck
>> >> >
>> >> >
>> >> > -----Original Message-----
>> >> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
>> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>> >> > To: [EMAIL PROTECTED]
>> >> > Subject: Managing space in Master Node
>> >> >
>> >> > Hello Gurus,
>> >> >
>> >> > I am managing a Hadoop Cluster to run some experiments.
>> >> >
>> >> > The issue I am continuously facing is that the Master Node runs out
>> >> > of
>> >> > disk
>> >> > space due to logs and data files.
>> >> >
>> >> > I can monitor and delete log files. However, I cannot delete the HDFS
>> >> > data.
>> >> >
>> >> > Thus, is there a way to force Hadoop not to save any HDFS data in the
>> >> > Master
>> >> > Node?
>> >> >
>> >> > Then I can use my master to handle the metadata only and store the
>> >> > logs.
>> >> >
>> >> > Thanks & regards
>> >> > Arko
>> >> > SDL Enterprise Technologies, Inc. - all rights reserved.  The
>> >> > information contained in this email may be confidential and/or
>> >> > legally
>> >> > privileged. It has been sent for the sole use of the intended
>> >> > recipient(s).
>> >> > If you are not the intended recipient of this mail, you are hereby
>> >> > notified
>> >> > that any unauthorized review, use, disclosure, dissemination,
>> >> > distribution,
>> >> > or copying of this communication, or any of its contents, is strictly
>> >> > prohibited. If you have received this communication in error, please
>> >> > reply
>> >> > to the sender and destroy all copies of the message.
>> >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA
>> >> > 01880,
>> >> > USA
>> >> >
>> >
>> >
>
>

The best way is to decommission the DN so the blocks get replicated to the
other nodes in the cluster. Manually copying the data is not recommended. See
if this works for you:
http://wiki.apache.org/hadoop/FAQ#I_want_to_make_a_large_cluster_smaller_by_t
aking_out_a_bunch_of_nodes_simultaneously._How_can_this_be_done.3F

-----Original Message-----
From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, February 13, 2013 12:13 PM
To: [EMAIL PROTECTED]
Subject: Re: Managing space in Master Node

Thanks!
However, in that case, how do I handle the current data that is already in
NN?
Do I have to reformat HDFS?
Thanks & regards
Arko

On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Just don't run the DN daemon on that machine. Dedicate that machine to NN.
> Remove the hostname of the NN machine from the 'slaves' file.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> Thanks for the help!
>>
>> However, I am still unsure about how to "turn off" the datanode
>> feature on the NN.
>>
>> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>> a directory.
>>
>> Should I just comment out that property? What would happen to the
>> current data that there is in the Master? Will it get removed
>> automatically?
>>
>> Thanks & regards
>> Arko
>>
>>
>>
>> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> wrote:
>> > You can specify the logging level as specified by Charles. But
>> > turning logs off is never a good idea. Logs are really helpful in
>> > problem diagnosis, which are eventual.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> > <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Yeah, my NameNode is also seconding as a DataNode.
>> >>
>> >> I would like to "turn off" this feature.
>> >>
>> >> Request help regarding the same.
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]> wrote:
>> >> > Hi Arko. Sounds like you may be running a DataNode on the
>> >> > NameNode which is not recommended practice. Normally, the only
>> >> > files the NN stores are the image and edits files. It does not
>> >> > store any actual HDFS data. If you must run a DN on the NN, try
>> >> > turning down the logging in
>> >> > /conf/log4j.properties:
>> >> >
>> >> > #hadoop.root.logger=INFO,console
>> >> > #hadoop.root.logger=WARN,console
>> >> > hadoop.root.logger=ERROR,console
>> >> >
>> >> > Depending on the logging information you require, of course.
>> >> >
>> >> > -Chuck
>> >> >
>> >> >
>> >> > -----Original Message-----
>> >> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
>> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>> >> > To: [EMAIL PROTECTED]
>> >> > Subject: Managing space in Master Node
>> >> >
>> >> > Hello Gurus,
>> >> >
>> >> > I am managing a Hadoop Cluster to run some experiments.
>> >> >
>> >> > The issue I am continuously facing is that the Master Node runs
>> >> > out of disk space due to logs and data files.
>> >> >
>> >> > I can monitor and delete log files. However, I cannot delete the
>> >> > HDFS data.
>> >> >
>> >> > Thus, is there a way to force Hadoop not to save any HDFS data
>> >> > in the Master Node?
>> >> >
>> >> > Then I can use my master to handle the metadata only and store
>> >> > the logs.
>> >> >
>> >> > Thanks & regards
>> >> > Arko
>> >> > SDL Enterprise Technologies, Inc. - all rights reserved.  The
>> >> > information contained in this email may be confidential and/or
>> >> > legally privileged. It has been sent for the sole use of the
>> >> > intended recipient(s).
>> >> > If you are not the intended recipient of this mail, you are
SDL Enterprise Technologies, Inc. - all rights reserved.  The information contained in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not the intended recipient of this mail, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please reply to the sender and destroy all copies of the message.
Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA

No, copy this directory to the new location of your choice.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee <
[EMAIL PROTECTED]> wrote:

> Thanks!
> However, in that case, how do I handle the current data that is already in
> NN?
> Do I have to reformat HDFS?
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> > Just don't run the DN daemon on that machine. Dedicate that machine to
> NN.
> > Remove the hostname of the NN machine from the 'slaves' file.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> > <[EMAIL PROTECTED]> wrote:
> >>
> >> Hi,
> >>
> >> Thanks for the help!
> >>
> >> However, I am still unsure about how to "turn off" the datanode
> >> feature on the NN.
> >>
> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
> >> a directory.
> >>
> >> Should I just comment out that property? What would happen to the
> >> current data that there is in the Master? Will it get removed
> >> automatically?
> >>
> >> Thanks & regards
> >> Arko
> >>
> >>
> >>
> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> >> wrote:
> >> > You can specify the logging level as specified by Charles. But turning
> >> > logs
> >> > off is never a good idea. Logs are really helpful in problem
> diagnosis,
> >> > which are eventual.
> >> >
> >> > Warm Regards,
> >> > Tariq
> >> > https://mtariq.jux.com/
> >> > cloudfront.blogspot.com
> >> >
> >> >
> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> >> > <[EMAIL PROTECTED]> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> Yeah, my NameNode is also seconding as a DataNode.
> >> >>
> >> >> I would like to "turn off" this feature.
> >> >>
> >> >> Request help regarding the same.
> >> >>
> >> >> Thanks & regards
> >> >> Arko
> >> >>
> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]>
> wrote:
> >> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
> >> >> > which
> >> >> > is
> >> >> > not recommended practice. Normally, the only files the NN stores
> are
> >> >> > the
> >> >> > image and edits files. It does not store any actual HDFS data. If
> you
> >> >> > must
> >> >> > run a DN on the NN, try turning down the logging in
> >> >> > /conf/log4j.properties:
> >> >> >
> >> >> > #hadoop.root.logger=INFO,console
> >> >> > #hadoop.root.logger=WARN,console
> >> >> > hadoop.root.logger=ERROR,console
> >> >> >
> >> >> > Depending on the logging information you require, of course.
> >> >> >
> >> >> > -Chuck
> >> >> >
> >> >> >
> >> >> > -----Original Message-----
> >> >> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
> >> >> > To: [EMAIL PROTECTED]
> >> >> > Subject: Managing space in Master Node
> >> >> >
> >> >> > Hello Gurus,
> >> >> >
> >> >> > I am managing a Hadoop Cluster to run some experiments.
> >> >> >
> >> >> > The issue I am continuously facing is that the Master Node runs out
> >> >> > of
> >> >> > disk
> >> >> > space due to logs and data files.
> >> >> >
> >> >> > I can monitor and delete log files. However, I cannot delete the
> HDFS
> >> >> > data.
> >> >> >
> >> >> > Thus, is there a way to force Hadoop not to save any HDFS data in
> the
> >> >> > Master
> >> >> > Node?
> >> >> >
> >> >> > Then I can use my master to handle the metadata only and store the
> >> >> > logs.
> >> >> >
> >> >> > Thanks & regards
> >> >> > Arko
> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved.  The
> >> >> > information contained in this email may be confidential and/or
> >> >> > legally
> >> >> > privileged. It has been sent for the sole use of the intended
> >> >> > recipient(s).
> >> >> > If you are not the intended recipient of this mail, you are hereby

Hi,

But the new location will also be the same NN physical server.
So how does that help?

I would like to clear up the space in the master by removing all the
HDFS data to the slave nodes.

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> No, copy this directory to the new location of your choice.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>>
>> Thanks!
>> However, in that case, how do I handle the current data that is already in
>> NN?
>> Do I have to reformat HDFS?
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> wrote:
>> > Just don't run the DN daemon on that machine. Dedicate that machine to
>> > NN.
>> > Remove the hostname of the NN machine from the 'slaves' file.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>> > <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Thanks for the help!
>> >>
>> >> However, I am still unsure about how to "turn off" the datanode
>> >> feature on the NN.
>> >>
>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>> >> a directory.
>> >>
>> >> Should I just comment out that property? What would happen to the
>> >> current data that there is in the Master? Will it get removed
>> >> automatically?
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >>
>> >>
>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > You can specify the logging level as specified by Charles. But
>> >> > turning
>> >> > logs
>> >> > off is never a good idea. Logs are really helpful in problem
>> >> > diagnosis,
>> >> > which are eventual.
>> >> >
>> >> > Warm Regards,
>> >> > Tariq
>> >> > https://mtariq.jux.com/
>> >> > cloudfront.blogspot.com
>> >> >
>> >> >
>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> >> > <[EMAIL PROTECTED]> wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>> >> >>
>> >> >> I would like to "turn off" this feature.
>> >> >>
>> >> >> Request help regarding the same.
>> >> >>
>> >> >> Thanks & regards
>> >> >> Arko
>> >> >>
>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]>
>> >> >> wrote:
>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
>> >> >> > which
>> >> >> > is
>> >> >> > not recommended practice. Normally, the only files the NN stores
>> >> >> > are
>> >> >> > the
>> >> >> > image and edits files. It does not store any actual HDFS data. If
>> >> >> > you
>> >> >> > must
>> >> >> > run a DN on the NN, try turning down the logging in
>> >> >> > /conf/log4j.properties:
>> >> >> >
>> >> >> > #hadoop.root.logger=INFO,console
>> >> >> > #hadoop.root.logger=WARN,console
>> >> >> > hadoop.root.logger=ERROR,console
>> >> >> >
>> >> >> > Depending on the logging information you require, of course.
>> >> >> >
>> >> >> > -Chuck
>> >> >> >
>> >> >> >
>> >> >> > -----Original Message-----
>> >> >> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>> >> >> > To: [EMAIL PROTECTED]
>> >> >> > Subject: Managing space in Master Node
>> >> >> >
>> >> >> > Hello Gurus,
>> >> >> >
>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>> >> >> >
>> >> >> > The issue I am continuously facing is that the Master Node runs
>> >> >> > out
>> >> >> > of
>> >> >> > disk
>> >> >> > space due to logs and data files.
>> >> >> >
>> >> >> > I can monitor and delete log files. However, I cannot delete the
>> >> >> > HDFS
>> >> >> > data.
>> >> >> >
>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS data in
>> >> >> > the
>> >> >> > Master
>> >> >> > Node?
>> >> >> >

Thanks! Let me study and find out if it works for me.
Thanks & regards
Arko

On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> But the new location will also be the same NN physical server.
> So how does that help?
>
> I would like to clear up the space in the master by removing all the
> HDFS data to the slave nodes.
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>> No, copy this directory to the new location of your choice.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>> <[EMAIL PROTECTED]> wrote:
>>>
>>> Thanks!
>>> However, in that case, how do I handle the current data that is already in
>>> NN?
>>> Do I have to reformat HDFS?
>>> Thanks & regards
>>> Arko
>>>
>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>> wrote:
>>> > Just don't run the DN daemon on that machine. Dedicate that machine to
>>> > NN.
>>> > Remove the hostname of the NN machine from the 'slaves' file.
>>> >
>>> > Warm Regards,
>>> > Tariq
>>> > https://mtariq.jux.com/
>>> > cloudfront.blogspot.com
>>> >
>>> >
>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>>> > <[EMAIL PROTECTED]> wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> Thanks for the help!
>>> >>
>>> >> However, I am still unsure about how to "turn off" the datanode
>>> >> feature on the NN.
>>> >>
>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>>> >> a directory.
>>> >>
>>> >> Should I just comment out that property? What would happen to the
>>> >> current data that there is in the Master? Will it get removed
>>> >> automatically?
>>> >>
>>> >> Thanks & regards
>>> >> Arko
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>> >> wrote:
>>> >> > You can specify the logging level as specified by Charles. But
>>> >> > turning
>>> >> > logs
>>> >> > off is never a good idea. Logs are really helpful in problem
>>> >> > diagnosis,
>>> >> > which are eventual.
>>> >> >
>>> >> > Warm Regards,
>>> >> > Tariq
>>> >> > https://mtariq.jux.com/
>>> >> > cloudfront.blogspot.com
>>> >> >
>>> >> >
>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>>> >> > <[EMAIL PROTECTED]> wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>>> >> >>
>>> >> >> I would like to "turn off" this feature.
>>> >> >>
>>> >> >> Request help regarding the same.
>>> >> >>
>>> >> >> Thanks & regards
>>> >> >> Arko
>>> >> >>
>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]>
>>> >> >> wrote:
>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
>>> >> >> > which
>>> >> >> > is
>>> >> >> > not recommended practice. Normally, the only files the NN stores
>>> >> >> > are
>>> >> >> > the
>>> >> >> > image and edits files. It does not store any actual HDFS data. If
>>> >> >> > you
>>> >> >> > must
>>> >> >> > run a DN on the NN, try turning down the logging in
>>> >> >> > /conf/log4j.properties:
>>> >> >> >
>>> >> >> > #hadoop.root.logger=INFO,console
>>> >> >> > #hadoop.root.logger=WARN,console
>>> >> >> > hadoop.root.logger=ERROR,console
>>> >> >> >
>>> >> >> > Depending on the logging information you require, of course.
>>> >> >> >
>>> >> >> > -Chuck
>>> >> >> >
>>> >> >> >
>>> >> >> > -----Original Message-----
>>> >> >> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>>> >> >> > To: [EMAIL PROTECTED]
>>> >> >> > Subject: Managing space in Master Node
>>> >> >> >
>>> >> >> > Hello Gurus,
>>> >> >> >
>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>>> >> >> >
>>> >> >> > The issue I am continuously facing is that the Master Node runs
>>> >> >> > out
>>> >> >> > of
>>> >> >> > disk

Hi,

I am not sure the provided link matches my situation.

That is because, I am not trying to exclude any node but just trying
to stop the NN act as a DataNode.

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
<[EMAIL PROTECTED]> wrote:
> Thanks! Let me study and find out if it works for me.
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> But the new location will also be the same NN physical server.
>> So how does that help?
>>
>> I would like to clear up the space in the master by removing all the
>> HDFS data to the slave nodes.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>>> No, copy this directory to the new location of your choice.
>>>
>>> Warm Regards,
>>> Tariq
>>> https://mtariq.jux.com/
>>> cloudfront.blogspot.com
>>>
>>>
>>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>>> <[EMAIL PROTECTED]> wrote:
>>>>
>>>> Thanks!
>>>> However, in that case, how do I handle the current data that is already in
>>>> NN?
>>>> Do I have to reformat HDFS?
>>>> Thanks & regards
>>>> Arko
>>>>
>>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>>> wrote:
>>>> > Just don't run the DN daemon on that machine. Dedicate that machine to
>>>> > NN.
>>>> > Remove the hostname of the NN machine from the 'slaves' file.
>>>> >
>>>> > Warm Regards,
>>>> > Tariq
>>>> > https://mtariq.jux.com/
>>>> > cloudfront.blogspot.com
>>>> >
>>>> >
>>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>>>> > <[EMAIL PROTECTED]> wrote:
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> Thanks for the help!
>>>> >>
>>>> >> However, I am still unsure about how to "turn off" the datanode
>>>> >> feature on the NN.
>>>> >>
>>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>>>> >> a directory.
>>>> >>
>>>> >> Should I just comment out that property? What would happen to the
>>>> >> current data that there is in the Master? Will it get removed
>>>> >> automatically?
>>>> >>
>>>> >> Thanks & regards
>>>> >> Arko
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>>> >> wrote:
>>>> >> > You can specify the logging level as specified by Charles. But
>>>> >> > turning
>>>> >> > logs
>>>> >> > off is never a good idea. Logs are really helpful in problem
>>>> >> > diagnosis,
>>>> >> > which are eventual.
>>>> >> >
>>>> >> > Warm Regards,
>>>> >> > Tariq
>>>> >> > https://mtariq.jux.com/
>>>> >> > cloudfront.blogspot.com
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>>>> >> > <[EMAIL PROTECTED]> wrote:
>>>> >> >>
>>>> >> >> Hi,
>>>> >> >>
>>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>>>> >> >>
>>>> >> >> I would like to "turn off" this feature.
>>>> >> >>
>>>> >> >> Request help regarding the same.
>>>> >> >>
>>>> >> >> Thanks & regards
>>>> >> >> Arko
>>>> >> >>
>>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]>
>>>> >> >> wrote:
>>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
>>>> >> >> > which
>>>> >> >> > is
>>>> >> >> > not recommended practice. Normally, the only files the NN stores
>>>> >> >> > are
>>>> >> >> > the
>>>> >> >> > image and edits files. It does not store any actual HDFS data. If
>>>> >> >> > you
>>>> >> >> > must
>>>> >> >> > run a DN on the NN, try turning down the logging in
>>>> >> >> > /conf/log4j.properties:
>>>> >> >> >
>>>> >> >> > #hadoop.root.logger=INFO,console
>>>> >> >> > #hadoop.root.logger=WARN,console
>>>> >> >> > hadoop.root.logger=ERROR,console
>>>> >> >> >
>>>> >> >> > Depending on the logging information you require, of course.
>>>> >> >> >
>>>> >> >> > -Chuck
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > -----Original Message-----
>>>> >> >> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]

Stopping DN on one machine means excluding it. What exactly is the
requirement? How many nodes do you have right now?

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am not sure the provided link matches my situation.
>
> That is because, I am not trying to exclude any node but just trying
> to stop the NN act as a DataNode.
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
> > Thanks! Let me study and find out if it works for me.
> > Thanks & regards
> > Arko
> >
> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> > <[EMAIL PROTECTED]> wrote:
> >> Hi,
> >>
> >> But the new location will also be the same NN physical server.
> >> So how does that help?
> >>
> >> I would like to clear up the space in the master by removing all the
> >> HDFS data to the slave nodes.
> >>
> >> Thanks & regards
> >> Arko
> >>
> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> >>> No, copy this directory to the new location of your choice.
> >>>
> >>> Warm Regards,
> >>> Tariq
> >>> https://mtariq.jux.com/
> >>> cloudfront.blogspot.com
> >>>
> >>>
> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> >>> <[EMAIL PROTECTED]> wrote:
> >>>>
> >>>> Thanks!
> >>>> However, in that case, how do I handle the current data that is
> already in
> >>>> NN?
> >>>> Do I have to reformat HDFS?
> >>>> Thanks & regards
> >>>> Arko
> >>>>
> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> >>>> wrote:
> >>>> > Just don't run the DN daemon on that machine. Dedicate that machine
> to
> >>>> > NN.
> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
> >>>> >
> >>>> > Warm Regards,
> >>>> > Tariq
> >>>> > https://mtariq.jux.com/
> >>>> > cloudfront.blogspot.com
> >>>> >
> >>>> >
> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> >>>> > <[EMAIL PROTECTED]> wrote:
> >>>> >>
> >>>> >> Hi,
> >>>> >>
> >>>> >> Thanks for the help!
> >>>> >>
> >>>> >> However, I am still unsure about how to "turn off" the datanode
> >>>> >> feature on the NN.
> >>>> >>
> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed
> to
> >>>> >> a directory.
> >>>> >>
> >>>> >> Should I just comment out that property? What would happen to the
> >>>> >> current data that there is in the Master? Will it get removed
> >>>> >> automatically?
> >>>> >>
> >>>> >> Thanks & regards
> >>>> >> Arko
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <
> [EMAIL PROTECTED]>
> >>>> >> wrote:
> >>>> >> > You can specify the logging level as specified by Charles. But
> >>>> >> > turning
> >>>> >> > logs
> >>>> >> > off is never a good idea. Logs are really helpful in problem
> >>>> >> > diagnosis,
> >>>> >> > which are eventual.
> >>>> >> >
> >>>> >> > Warm Regards,
> >>>> >> > Tariq
> >>>> >> > https://mtariq.jux.com/
> >>>> >> > cloudfront.blogspot.com
> >>>> >> >
> >>>> >> >
> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> >>>> >> > <[EMAIL PROTECTED]> wrote:
> >>>> >> >>
> >>>> >> >> Hi,
> >>>> >> >>
> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
> >>>> >> >>
> >>>> >> >> I would like to "turn off" this feature.
> >>>> >> >>
> >>>> >> >> Request help regarding the same.
> >>>> >> >>
> >>>> >> >> Thanks & regards
> >>>> >> >> Arko
> >>>> >> >>
> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]>
> >>>> >> >> wrote:
> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the
> NameNode
> >>>> >> >> > which
> >>>> >> >> > is
> >>>> >> >> > not recommended practice. Normally, the only files the NN
> stores
> >>>> >> >> > are
> >>>> >> >> > the
> >>>> >> >> > image and edits files. It does not store any actual HDFS
> data. If
> >>>> >>

Hi,

I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
(running DN and Task Tracker).

My issue is that the my NN is getting filled up pretty fast as the NN
acts as a DN.

I want my NN not to function as the DN.

Does including the IP for the NN in the "dfs.hosts.exclude" and
running "$ hadoop dfsadmin -refreshNodes" solve my problem?

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Stopping DN on one machine means excluding it. What exactly is the
> requirement? How many nodes do you have right now?
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> I am not sure the provided link matches my situation.
>>
>> That is because, I am not trying to exclude any node but just trying
>> to stop the NN act as a DataNode.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
>> <[EMAIL PROTECTED]> wrote:
>> > Thanks! Let me study and find out if it works for me.
>> > Thanks & regards
>> > Arko
>> >
>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
>> > <[EMAIL PROTECTED]> wrote:
>> >> Hi,
>> >>
>> >> But the new location will also be the same NN physical server.
>> >> So how does that help?
>> >>
>> >> I would like to clear up the space in the master by removing all the
>> >> HDFS data to the slave nodes.
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> >> wrote:
>> >>> No, copy this directory to the new location of your choice.
>> >>>
>> >>> Warm Regards,
>> >>> Tariq
>> >>> https://mtariq.jux.com/
>> >>> cloudfront.blogspot.com
>> >>>
>> >>>
>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>> >>> <[EMAIL PROTECTED]> wrote:
>> >>>>
>> >>>> Thanks!
>> >>>> However, in that case, how do I handle the current data that is
>> >>>> already in
>> >>>> NN?
>> >>>> Do I have to reformat HDFS?
>> >>>> Thanks & regards
>> >>>> Arko
>> >>>>
>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> >>>> wrote:
>> >>>> > Just don't run the DN daemon on that machine. Dedicate that machine
>> >>>> > to
>> >>>> > NN.
>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
>> >>>> >
>> >>>> > Warm Regards,
>> >>>> > Tariq
>> >>>> > https://mtariq.jux.com/
>> >>>> > cloudfront.blogspot.com
>> >>>> >
>> >>>> >
>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>> >>>> > <[EMAIL PROTECTED]> wrote:
>> >>>> >>
>> >>>> >> Hi,
>> >>>> >>
>> >>>> >> Thanks for the help!
>> >>>> >>
>> >>>> >> However, I am still unsure about how to "turn off" the datanode
>> >>>> >> feature on the NN.
>> >>>> >>
>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed
>> >>>> >> to
>> >>>> >> a directory.
>> >>>> >>
>> >>>> >> Should I just comment out that property? What would happen to the
>> >>>> >> current data that there is in the Master? Will it get removed
>> >>>> >> automatically?
>> >>>> >>
>> >>>> >> Thanks & regards
>> >>>> >> Arko
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
>> >>>> >> <[EMAIL PROTECTED]>
>> >>>> >> wrote:
>> >>>> >> > You can specify the logging level as specified by Charles. But
>> >>>> >> > turning
>> >>>> >> > logs
>> >>>> >> > off is never a good idea. Logs are really helpful in problem
>> >>>> >> > diagnosis,
>> >>>> >> > which are eventual.
>> >>>> >> >
>> >>>> >> > Warm Regards,
>> >>>> >> > Tariq
>> >>>> >> > https://mtariq.jux.com/
>> >>>> >> > cloudfront.blogspot.com
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> >>>> >> > <[EMAIL PROTECTED]> wrote:
>> >>>> >> >>
>> >>>> >> >> Hi,
>> >>>> >> >>
>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>> >>>> >> >>
>>

Hi,

I see that the NameNode IP is present in conf/slaves file.

Will removing the IP of the NN from the conf/slaves file and
restarting hadoop help?

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
> (running DN and Task Tracker).
>
> My issue is that the my NN is getting filled up pretty fast as the NN
> acts as a DN.
>
> I want my NN not to function as the DN.
>
> Does including the IP for the NN in the "dfs.hosts.exclude" and
> running "$ hadoop dfsadmin -refreshNodes" solve my problem?
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>> Stopping DN on one machine means excluding it. What exactly is the
>> requirement? How many nodes do you have right now?
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
>> <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi,
>>>
>>> I am not sure the provided link matches my situation.
>>>
>>> That is because, I am not trying to exclude any node but just trying
>>> to stop the NN act as a DataNode.
>>>
>>> Thanks & regards
>>> Arko
>>>
>>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
>>> <[EMAIL PROTECTED]> wrote:
>>> > Thanks! Let me study and find out if it works for me.
>>> > Thanks & regards
>>> > Arko
>>> >
>>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
>>> > <[EMAIL PROTECTED]> wrote:
>>> >> Hi,
>>> >>
>>> >> But the new location will also be the same NN physical server.
>>> >> So how does that help?
>>> >>
>>> >> I would like to clear up the space in the master by removing all the
>>> >> HDFS data to the slave nodes.
>>> >>
>>> >> Thanks & regards
>>> >> Arko
>>> >>
>>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>> >> wrote:
>>> >>> No, copy this directory to the new location of your choice.
>>> >>>
>>> >>> Warm Regards,
>>> >>> Tariq
>>> >>> https://mtariq.jux.com/
>>> >>> cloudfront.blogspot.com
>>> >>>
>>> >>>
>>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>>> >>> <[EMAIL PROTECTED]> wrote:
>>> >>>>
>>> >>>> Thanks!
>>> >>>> However, in that case, how do I handle the current data that is
>>> >>>> already in
>>> >>>> NN?
>>> >>>> Do I have to reformat HDFS?
>>> >>>> Thanks & regards
>>> >>>> Arko
>>> >>>>
>>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>> >>>> wrote:
>>> >>>> > Just don't run the DN daemon on that machine. Dedicate that machine
>>> >>>> > to
>>> >>>> > NN.
>>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
>>> >>>> >
>>> >>>> > Warm Regards,
>>> >>>> > Tariq
>>> >>>> > https://mtariq.jux.com/
>>> >>>> > cloudfront.blogspot.com
>>> >>>> >
>>> >>>> >
>>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>>> >>>> > <[EMAIL PROTECTED]> wrote:
>>> >>>> >>
>>> >>>> >> Hi,
>>> >>>> >>
>>> >>>> >> Thanks for the help!
>>> >>>> >>
>>> >>>> >> However, I am still unsure about how to "turn off" the datanode
>>> >>>> >> feature on the NN.
>>> >>>> >>
>>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed
>>> >>>> >> to
>>> >>>> >> a directory.
>>> >>>> >>
>>> >>>> >> Should I just comment out that property? What would happen to the
>>> >>>> >> current data that there is in the Master? Will it get removed
>>> >>>> >> automatically?
>>> >>>> >>
>>> >>>> >> Thanks & regards
>>> >>>> >> Arko
>>> >>>> >>
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
>>> >>>> >> <[EMAIL PROTECTED]>
>>> >>>> >> wrote:
>>> >>>> >> > You can specify the logging level as specified by Charles. But
>>> >>>> >> > turning
>>> >>>> >> > logs
>>> >>>> >> > off is never a good idea. Logs are really helpful in problem
>>> >>>> >> > diagnosis,
>>> >>>> >> > which are eventual.
>>>

It should.

BTW what's the problem with decommissioning?It's the correct way of
removing the DN, as said by Charles earlier. Create the exclude file and
add the property in the hdfs-site.xml file.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I see that the NameNode IP is present in conf/slaves file.
>
> Will removing the IP of the NN from the conf/slaves file and
> restarting hadoop help?
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
> > (running DN and Task Tracker).
> >
> > My issue is that the my NN is getting filled up pretty fast as the NN
> > acts as a DN.
> >
> > I want my NN not to function as the DN.
> >
> > Does including the IP for the NN in the "dfs.hosts.exclude" and
> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
> >
> > Thanks & regards
> > Arko
> >
> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> >> Stopping DN on one machine means excluding it. What exactly is the
> >> requirement? How many nodes do you have right now?
> >>
> >> Warm Regards,
> >> Tariq
> >> https://mtariq.jux.com/
> >> cloudfront.blogspot.com
> >>
> >>
> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> >> <[EMAIL PROTECTED]> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am not sure the provided link matches my situation.
> >>>
> >>> That is because, I am not trying to exclude any node but just trying
> >>> to stop the NN act as a DataNode.
> >>>
> >>> Thanks & regards
> >>> Arko
> >>>
> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
> >>> <[EMAIL PROTECTED]> wrote:
> >>> > Thanks! Let me study and find out if it works for me.
> >>> > Thanks & regards
> >>> > Arko
> >>> >
> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> >>> > <[EMAIL PROTECTED]> wrote:
> >>> >> Hi,
> >>> >>
> >>> >> But the new location will also be the same NN physical server.
> >>> >> So how does that help?
> >>> >>
> >>> >> I would like to clear up the space in the master by removing all the
> >>> >> HDFS data to the slave nodes.
> >>> >>
> >>> >> Thanks & regards
> >>> >> Arko
> >>> >>
> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]
> >
> >>> >> wrote:
> >>> >>> No, copy this directory to the new location of your choice.
> >>> >>>
> >>> >>> Warm Regards,
> >>> >>> Tariq
> >>> >>> https://mtariq.jux.com/
> >>> >>> cloudfront.blogspot.com
> >>> >>>
> >>> >>>
> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> >>> >>> <[EMAIL PROTECTED]> wrote:
> >>> >>>>
> >>> >>>> Thanks!
> >>> >>>> However, in that case, how do I handle the current data that is
> >>> >>>> already in
> >>> >>>> NN?
> >>> >>>> Do I have to reformat HDFS?
> >>> >>>> Thanks & regards
> >>> >>>> Arko
> >>> >>>>
> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <
> [EMAIL PROTECTED]>
> >>> >>>> wrote:
> >>> >>>> > Just don't run the DN daemon on that machine. Dedicate that
> machine
> >>> >>>> > to
> >>> >>>> > NN.
> >>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
> >>> >>>> >
> >>> >>>> > Warm Regards,
> >>> >>>> > Tariq
> >>> >>>> > https://mtariq.jux.com/
> >>> >>>> > cloudfront.blogspot.com
> >>> >>>> >
> >>> >>>> >
> >>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> >>> >>>> > <[EMAIL PROTECTED]> wrote:
> >>> >>>> >>
> >>> >>>> >> Hi,
> >>> >>>> >>
> >>> >>>> >> Thanks for the help!
> >>> >>>> >>
> >>> >>>> >> However, I am still unsure about how to "turn off" the datanode
> >>> >>>> >> feature on the NN.
> >>> >>>> >>
> >>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is
> pointed
> >>> >>>> >> to
> >>> >>>> >> a directory.
> >>> >>>> >>
> >>> >>>> >> Should I just comment out that property? What would happen to

Hi,

I was just wondering if it is okay to exclude the NN IP.

I hope it won't create any "side effects".

This is because I don't want to actually decommission the NN node, but
just make it stop working as a DataNode.

However, if you are sure there are no problems, I will go ahead and
make the changes.

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 5:43 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> It should.
>
> BTW what's the problem with decommissioning?It's the correct way of removing
> the DN, as said by Charles earlier. Create the exclude file and add the
> property in the hdfs-site.xml file.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> I see that the NameNode IP is present in conf/slaves file.
>>
>> Will removing the IP of the NN from the conf/slaves file and
>> restarting hadoop help?
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
>> <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> >
>> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
>> > (running DN and Task Tracker).
>> >
>> > My issue is that the my NN is getting filled up pretty fast as the NN
>> > acts as a DN.
>> >
>> > I want my NN not to function as the DN.
>> >
>> > Does including the IP for the NN in the "dfs.hosts.exclude" and
>> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
>> >
>> > Thanks & regards
>> > Arko
>> >
>> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> > wrote:
>> >> Stopping DN on one machine means excluding it. What exactly is the
>> >> requirement? How many nodes do you have right now?
>> >>
>> >> Warm Regards,
>> >> Tariq
>> >> https://mtariq.jux.com/
>> >> cloudfront.blogspot.com
>> >>
>> >>
>> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
>> >> <[EMAIL PROTECTED]> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I am not sure the provided link matches my situation.
>> >>>
>> >>> That is because, I am not trying to exclude any node but just trying
>> >>> to stop the NN act as a DataNode.
>> >>>
>> >>> Thanks & regards
>> >>> Arko
>> >>>
>> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
>> >>> <[EMAIL PROTECTED]> wrote:
>> >>> > Thanks! Let me study and find out if it works for me.
>> >>> > Thanks & regards
>> >>> > Arko
>> >>> >
>> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
>> >>> > <[EMAIL PROTECTED]> wrote:
>> >>> >> Hi,
>> >>> >>
>> >>> >> But the new location will also be the same NN physical server.
>> >>> >> So how does that help?
>> >>> >>
>> >>> >> I would like to clear up the space in the master by removing all
>> >>> >> the
>> >>> >> HDFS data to the slave nodes.
>> >>> >>
>> >>> >> Thanks & regards
>> >>> >> Arko
>> >>> >>
>> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq
>> >>> >> <[EMAIL PROTECTED]>
>> >>> >> wrote:
>> >>> >>> No, copy this directory to the new location of your choice.
>> >>> >>>
>> >>> >>> Warm Regards,
>> >>> >>> Tariq
>> >>> >>> https://mtariq.jux.com/
>> >>> >>> cloudfront.blogspot.com
>> >>> >>>
>> >>> >>>
>> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>> >>> >>> <[EMAIL PROTECTED]> wrote:
>> >>> >>>>
>> >>> >>>> Thanks!
>> >>> >>>> However, in that case, how do I handle the current data that is
>> >>> >>>> already in
>> >>> >>>> NN?
>> >>> >>>> Do I have to reformat HDFS?
>> >>> >>>> Thanks & regards
>> >>> >>>> Arko
>> >>> >>>>
>> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq
>> >>> >>>> <[EMAIL PROTECTED]>
>> >>> >>>> wrote:
>> >>> >>>> > Just don't run the DN daemon on that machine. Dedicate that
>> >>> >>>> > machine
>> >>> >>>> > to
>> >>> >>>> > NN.
>> >>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
>> >>> >>>> >
>> >>> >>>> > Warm Regards,
>> >>> >>>> > Tariq
>> >>> >>>> > https://mtariq.jux.com/
>> >

Decommissioning is for removing DNs and not the NN.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 5:18 AM, Arko Provo Mukherjee <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I was just wondering if it is okay to exclude the NN IP.
>
> I hope it won't create any "side effects".
>
> This is because I don't want to actually decommission the NN node, but
> just make it stop working as a DataNode.
>
> However, if you are sure there are no problems, I will go ahead and
> make the changes.
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 5:43 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> > It should.
> >
> > BTW what's the problem with decommissioning?It's the correct way of
> removing
> > the DN, as said by Charles earlier. Create the exclude file and add the
> > property in the hdfs-site.xml file.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee
> > <[EMAIL PROTECTED]> wrote:
> >>
> >> Hi,
> >>
> >> I see that the NameNode IP is present in conf/slaves file.
> >>
> >> Will removing the IP of the NN from the conf/slaves file and
> >> restarting hadoop help?
> >>
> >> Thanks & regards
> >> Arko
> >>
> >> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
> >> <[EMAIL PROTECTED]> wrote:
> >> > Hi,
> >> >
> >> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
> >> > (running DN and Task Tracker).
> >> >
> >> > My issue is that the my NN is getting filled up pretty fast as the NN
> >> > acts as a DN.
> >> >
> >> > I want my NN not to function as the DN.
> >> >
> >> > Does including the IP for the NN in the "dfs.hosts.exclude" and
> >> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
> >> >
> >> > Thanks & regards
> >> > Arko
> >> >
> >> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> >> > wrote:
> >> >> Stopping DN on one machine means excluding it. What exactly is the
> >> >> requirement? How many nodes do you have right now?
> >> >>
> >> >> Warm Regards,
> >> >> Tariq
> >> >> https://mtariq.jux.com/
> >> >> cloudfront.blogspot.com
> >> >>
> >> >>
> >> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> >> >> <[EMAIL PROTECTED]> wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> I am not sure the provided link matches my situation.
> >> >>>
> >> >>> That is because, I am not trying to exclude any node but just trying
> >> >>> to stop the NN act as a DataNode.
> >> >>>
> >> >>> Thanks & regards
> >> >>> Arko
> >> >>>
> >> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
> >> >>> <[EMAIL PROTECTED]> wrote:
> >> >>> > Thanks! Let me study and find out if it works for me.
> >> >>> > Thanks & regards
> >> >>> > Arko
> >> >>> >
> >> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> >> >>> > <[EMAIL PROTECTED]> wrote:
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> But the new location will also be the same NN physical server.
> >> >>> >> So how does that help?
> >> >>> >>
> >> >>> >> I would like to clear up the space in the master by removing all
> >> >>> >> the
> >> >>> >> HDFS data to the slave nodes.
> >> >>> >>
> >> >>> >> Thanks & regards
> >> >>> >> Arko
> >> >>> >>
> >> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq
> >> >>> >> <[EMAIL PROTECTED]>
> >> >>> >> wrote:
> >> >>> >>> No, copy this directory to the new location of your choice.
> >> >>> >>>
> >> >>> >>> Warm Regards,
> >> >>> >>> Tariq
> >> >>> >>> https://mtariq.jux.com/
> >> >>> >>> cloudfront.blogspot.com
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> >> >>> >>> <[EMAIL PROTECTED]> wrote:
> >> >>> >>>>
> >> >>> >>>> Thanks!
> >> >>> >>>> However, in that case, how do I handle the current data that is
> >> >>> >>>> already in
> >> >>> >>>> NN?
> >> >>> >>>> Do I have to reformat HDFS?
> >> >>> >>>> Thanks & regards

Sure, thanks!
One last question before I go ahead and make the changes.
My hdfs-site.xml doesn't have an entry for dfs.replication.
So can I just assume that the system is using the default of 3?
I ask this because if it is 1, then I will first have to increase it
so as to not loose data when I stop the NN being a DN.
Thanks & regards
Arko

On Wed, Feb 13, 2013 at 5:51 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Decommissioning is for removing DNs and not the NN.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 5:18 AM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> I was just wondering if it is okay to exclude the NN IP.
>>
>> I hope it won't create any "side effects".
>>
>> This is because I don't want to actually decommission the NN node, but
>> just make it stop working as a DataNode.
>>
>> However, if you are sure there are no problems, I will go ahead and
>> make the changes.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 5:43 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> wrote:
>> > It should.
>> >
>> > BTW what's the problem with decommissioning?It's the correct way of
>> > removing
>> > the DN, as said by Charles earlier. Create the exclude file and add the
>> > property in the hdfs-site.xml file.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee
>> > <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I see that the NameNode IP is present in conf/slaves file.
>> >>
>> >> Will removing the IP of the NN from the conf/slaves file and
>> >> restarting hadoop help?
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
>> >> <[EMAIL PROTECTED]> wrote:
>> >> > Hi,
>> >> >
>> >> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave
>> >> > Nodes
>> >> > (running DN and Task Tracker).
>> >> >
>> >> > My issue is that the my NN is getting filled up pretty fast as the NN
>> >> > acts as a DN.
>> >> >
>> >> > I want my NN not to function as the DN.
>> >> >
>> >> > Does including the IP for the NN in the "dfs.hosts.exclude" and
>> >> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
>> >> >
>> >> > Thanks & regards
>> >> > Arko
>> >> >
>> >> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> >> > wrote:
>> >> >> Stopping DN on one machine means excluding it. What exactly is the
>> >> >> requirement? How many nodes do you have right now?
>> >> >>
>> >> >> Warm Regards,
>> >> >> Tariq
>> >> >> https://mtariq.jux.com/
>> >> >> cloudfront.blogspot.com
>> >> >>
>> >> >>
>> >> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
>> >> >> <[EMAIL PROTECTED]> wrote:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> I am not sure the provided link matches my situation.
>> >> >>>
>> >> >>> That is because, I am not trying to exclude any node but just
>> >> >>> trying
>> >> >>> to stop the NN act as a DataNode.
>> >> >>>
>> >> >>> Thanks & regards
>> >> >>> Arko
>> >> >>>
>> >> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
>> >> >>> <[EMAIL PROTECTED]> wrote:
>> >> >>> > Thanks! Let me study and find out if it works for me.
>> >> >>> > Thanks & regards
>> >> >>> > Arko
>> >> >>> >
>> >> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
>> >> >>> > <[EMAIL PROTECTED]> wrote:
>> >> >>> >> Hi,
>> >> >>> >>
>> >> >>> >> But the new location will also be the same NN physical server.
>> >> >>> >> So how does that help?
>> >> >>> >>
>> >> >>> >> I would like to clear up the space in the master by removing all
>> >> >>> >> the
>> >> >>> >> HDFS data to the slave nodes.
>> >> >>> >>
>> >> >>> >> Thanks & regards
>> >> >>> >> Arko
>> >> >>> >>
>> >> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq
>> >> >>> >> <[EMAIL PROTECTED]>
>> >> >>> >> wrote:
>> >>

Yes.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 5:24 AM, Arko Provo Mukherjee <
[EMAIL PROTECTED]> wrote:

> Sure, thanks!
> One last question before I go ahead and make the changes.
> My hdfs-site.xml doesn't have an entry for dfs.replication.
> So can I just assume that the system is using the default of 3?
> I ask this because if it is 1, then I will first have to increase it
> so as to not loose data when I stop the NN being a DN.
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 5:51 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> > Decommissioning is for removing DNs and not the NN.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Thu, Feb 14, 2013 at 5:18 AM, Arko Provo Mukherjee
> > <[EMAIL PROTECTED]> wrote:
> >>
> >> Hi,
> >>
> >> I was just wondering if it is okay to exclude the NN IP.
> >>
> >> I hope it won't create any "side effects".
> >>
> >> This is because I don't want to actually decommission the NN node, but
> >> just make it stop working as a DataNode.
> >>
> >> However, if you are sure there are no problems, I will go ahead and
> >> make the changes.
> >>
> >> Thanks & regards
> >> Arko
> >>
> >> On Wed, Feb 13, 2013 at 5:43 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> >> wrote:
> >> > It should.
> >> >
> >> > BTW what's the problem with decommissioning?It's the correct way of
> >> > removing
> >> > the DN, as said by Charles earlier. Create the exclude file and add
> the
> >> > property in the hdfs-site.xml file.
> >> >
> >> > Warm Regards,
> >> > Tariq
> >> > https://mtariq.jux.com/
> >> > cloudfront.blogspot.com
> >> >
> >> >
> >> > On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee
> >> > <[EMAIL PROTECTED]> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> I see that the NameNode IP is present in conf/slaves file.
> >> >>
> >> >> Will removing the IP of the NN from the conf/slaves file and
> >> >> restarting hadoop help?
> >> >>
> >> >> Thanks & regards
> >> >> Arko
> >> >>
> >> >> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
> >> >> <[EMAIL PROTECTED]> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave
> >> >> > Nodes
> >> >> > (running DN and Task Tracker).
> >> >> >
> >> >> > My issue is that the my NN is getting filled up pretty fast as the
> NN
> >> >> > acts as a DN.
> >> >> >
> >> >> > I want my NN not to function as the DN.
> >> >> >
> >> >> > Does including the IP for the NN in the "dfs.hosts.exclude" and
> >> >> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
> >> >> >
> >> >> > Thanks & regards
> >> >> > Arko
> >> >> >
> >> >> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <
> [EMAIL PROTECTED]>
> >> >> > wrote:
> >> >> >> Stopping DN on one machine means excluding it. What exactly is the
> >> >> >> requirement? How many nodes do you have right now?
> >> >> >>
> >> >> >> Warm Regards,
> >> >> >> Tariq
> >> >> >> https://mtariq.jux.com/
> >> >> >> cloudfront.blogspot.com
> >> >> >>
> >> >> >>
> >> >> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> >> >> >> <[EMAIL PROTECTED]> wrote:
> >> >> >>>
> >> >> >>> Hi,
> >> >> >>>
> >> >> >>> I am not sure the provided link matches my situation.
> >> >> >>>
> >> >> >>> That is because, I am not trying to exclude any node but just
> >> >> >>> trying
> >> >> >>> to stop the NN act as a DataNode.
> >> >> >>>
> >> >> >>> Thanks & regards
> >> >> >>> Arko
> >> >> >>>
> >> >> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
> >> >> >>> <[EMAIL PROTECTED]> wrote:
> >> >> >>> > Thanks! Let me study and find out if it works for me.
> >> >> >>> > Thanks & regards
> >> >> >>> > Arko
> >> >> >>> >
> >> >> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> >> >> >>> > <[EMAIL PROTECTED]> wrote:
> >> >> >>> >> Hi,
> >> >> >>> >>
> >> >> >>> >> But the new location will also be the same NN physical server.

Hi Arko.

  Only thing you need to do is not running the TaskTracker and DataNode demons on your master machine. Ensure you do not have this on your slaves file con hadoop's config directory when you start the system. I'm supposing you are on Open Source release. For other distributions look at its documentation about how to remove those demons from your master node machine.
-----Original Message-----
From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
Sent: miércoles, 13 de febrero de 2013 20:32
To: [EMAIL PROTECTED]
Subject: Managing space in Master Node

Hello Gurus,

I am managing a Hadoop Cluster to run some experiments.

The issue I am continuously facing is that the Master Node runs out of
disk space due to logs and data files.

I can monitor and delete log files. However, I cannot delete the HDFS data.

Thus, is there a way to force Hadoop not to save any HDFS data in the
Master Node?

Then I can use my master to handle the metadata only and store the logs.

Thanks & regards
Arko
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited.

Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.

______________________________________________________________________________________

www.accenture.com

Thanks! I have already requested for downtime. Will do the changes soon!
Warm regards
Arko

On Thu, Feb 14, 2013 at 3:31 AM,  <[EMAIL PROTECTED]> wrote:
> Hi Arko.
>
>   Only thing you need to do is not running the TaskTracker and DataNode demons on your master machine. Ensure you do not have this on your slaves file con hadoop's config directory when you start the system. I'm supposing you are on Open Source release. For other distributions look at its documentation about how to remove those demons from your master node machine.
>
>
> -----Original Message-----
> From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
> Sent: miércoles, 13 de febrero de 2013 20:32
> To: [EMAIL PROTECTED]
> Subject: Managing space in Master Node
>
> Hello Gurus,
>
> I am managing a Hadoop Cluster to run some experiments.
>
> The issue I am continuously facing is that the Master Node runs out of
> disk space due to logs and data files.
>
> I can monitor and delete log files. However, I cannot delete the HDFS data.
>
> Thus, is there a way to force Hadoop not to save any HDFS data in the
> Master Node?
>
> Then I can use my master to handle the metadata only and store the logs.
>
> Thanks & regards
> Arko
>
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited.
>
> Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
>
> ______________________________________________________________________________________
>
> www.accenture.com
>

Hello Everyone - I have decommissioned my NN so that only the
JobTracker, NameNode and SecondaryNameNodes are running on it now.
I would like to thank everyone who helped me to perform this task!

I have just one more issue: The data files created in the Master Node
seem to be still there. in /hdfs/data/current
Now that this server is not being used as a DataNode, can I safely
delete the files under this folder to clear up space?

Thanks & regards
Arko

On Thu, Feb 14, 2013 at 11:48 AM, Arko Provo Mukherjee
<[EMAIL PROTECTED]> wrote:
> Thanks! I have already requested for downtime. Will do the changes soon!
> Warm regards
> Arko
>
> On Thu, Feb 14, 2013 at 3:31 AM,  <[EMAIL PROTECTED]> wrote:
>> Hi Arko.
>>
>>   Only thing you need to do is not running the TaskTracker and DataNode demons on your master machine. Ensure you do not have this on your slaves file con hadoop's config directory when you start the system. I'm supposing you are on Open Source release. For other distributions look at its documentation about how to remove those demons from your master node machine.
>>
>>
>> -----Original Message-----
>> From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
>> Sent: miércoles, 13 de febrero de 2013 20:32
>> To: [EMAIL PROTECTED]
>> Subject: Managing space in Master Node
>>
>> Hello Gurus,
>>
>> I am managing a Hadoop Cluster to run some experiments.
>>
>> The issue I am continuously facing is that the Master Node runs out of
>> disk space due to logs and data files.
>>
>> I can monitor and delete log files. However, I cannot delete the HDFS data.
>>
>> Thus, is there a way to force Hadoop not to save any HDFS data in the
>> Master Node?
>>
>> Then I can use my master to handle the metadata only and store the logs.
>>
>> Thanks & regards
>> Arko
>>
>>
>> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited.
>>
>> Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
>>
>> ______________________________________________________________________________________
>>
>> www.accenture.com
>>

Hey Arko. Glad to hear it worked out for you. Just make sure that your NN's
image and edit files aren't also (perhaps inadvertently) stored in that
directory structure. The NameNode image/edit dir contains the following:

drwxr-xr-x 2 hdfs hdfs 4096 Feb 14 17:48 current
drwxr-xr-x 2 hdfs hdfs 4096 Jul 17  2012 image
-rw-r--r-- 1 hdfs hdfs    0 Sep 27 12:28 in_use.lock
drwxr-xr-x 2 hdfs hdfs 4096 Sep 17 23:25 previous.checkpoint

whereas, the DN data dir should contain:

drwxr-xr-x  2 hdfs hdfs 57344 Feb 14 14:04 blocksBeingWritten
drwxr-xr-x 66 hdfs hdfs  8192 Jan 22 02:44 current
drwxr-xr-x  2 hdfs hdfs     6 Aug 10  2012 detach
-rw-r--r--  1 hdfs hdfs     0 Sep 27 11:19 in_use.lock
-rw-r--r--  1 hdfs hdfs   157 Aug 10  2012 storage
drwxr-xr-x  2 hdfs hdfs     6 Feb  4 17:34 tmp
It's a good practice to locate the JobTracker and SecondaryNameNode services
on a different machine than the NN. If you have an opportunity to do so, I
recommend getting a separate machine for those two services as well.  

-Chuck
-----Original Message-----
From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
Sent: Thursday, February 14, 2013 4:21 PM
To: [EMAIL PROTECTED]
Subject: Re: Managing space in Master Node

Hello Everyone - I have decommissioned my NN so that only the JobTracker,
NameNode and SecondaryNameNodes are running on it now.
I would like to thank everyone who helped me to perform this task!

I have just one more issue: The data files created in the Master Node seem to
be still there. in /hdfs/data/current Now that this server is not being used
as a DataNode, can I safely delete the files under this folder to clear up
space?

Thanks & regards
Arko

On Thu, Feb 14, 2013 at 11:48 AM, Arko Provo Mukherjee
<[EMAIL PROTECTED]> wrote:
> Thanks! I have already requested for downtime. Will do the changes soon!
> Warm regards
> Arko
>
> On Thu, Feb 14, 2013 at 3:31 AM,  <[EMAIL PROTECTED]> wrote:
>> Hi Arko.
>>
>>   Only thing you need to do is not running the TaskTracker and DataNode
demons on your master machine. Ensure you do not have this on your slaves
file con hadoop's config directory when you start the system. I'm supposing
you are on Open Source release. For other distributions look at its
documentation about how to remove those demons from your master node machine.
>>
>>
>> -----Original Message-----
>> From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
>> Sent: miércoles, 13 de febrero de 2013 20:32
>> To: [EMAIL PROTECTED]
>> Subject: Managing space in Master Node
>>
>> Hello Gurus,
>>
>> I am managing a Hadoop Cluster to run some experiments.
>>
>> The issue I am continuously facing is that the Master Node runs out
>> of disk space due to logs and data files.
>>
>> I can monitor and delete log files. However, I cannot delete the HDFS
data.
>>
>> Thus, is there a way to force Hadoop not to save any HDFS data in the
>> Master Node?
>>
>> Then I can use my master to handle the metadata only and store the logs.
>>
>> Thanks & regards
>> Arko
>>
>>
>> This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise private information. If you have
received it in error, please notify the sender immediately and delete the
original. Any other use of the e-mail by you is prohibited.
>>
>> Where allowed by local law, electronic communications with Accenture and
its affiliates, including e-mail and instant messaging (including content),
may be scanned by our systems for the purposes of information security and
assessment of internal compliance with Accenture policy.
>>
>> _____________________________________________________________________
>> _________________
>>
>> www.accenture.com
>>
SDL Enterprise Technologies, Inc. - all rights reserved.  The information contained in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not the intended recipient of this mail, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please reply to the sender and destroy all copies of the message.
Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA