Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Managing space in Master Node


+
Arko Provo Mukherjee 2013-02-13, 19:32
+
Mohammad Tariq 2013-02-13, 19:37
+
Charles Baker 2013-02-13, 19:38
+
Arko Provo Mukherjee 2013-02-13, 19:52
+
Mohammad Tariq 2013-02-13, 19:55
Copy link to this message
-
Re: Managing space in Master Node
Hi,

Thanks for the help!

However, I am still unsure about how to "turn off" the datanode
feature on the NN.

I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
a directory.

Should I just comment out that property? What would happen to the
current data that there is in the Master? Will it get removed
automatically?

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> You can specify the logging level as specified by Charles. But turning logs
> off is never a good idea. Logs are really helpful in problem diagnosis,
> which are eventual.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> Yeah, my NameNode is also seconding as a DataNode.
>>
>> I would like to "turn off" this feature.
>>
>> Request help regarding the same.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]> wrote:
>> > Hi Arko. Sounds like you may be running a DataNode on the NameNode which
>> > is
>> > not recommended practice. Normally, the only files the NN stores are the
>> > image and edits files. It does not store any actual HDFS data. If you
>> > must
>> > run a DN on the NN, try turning down the logging in
>> > /conf/log4j.properties:
>> >
>> > #hadoop.root.logger=INFO,console
>> > #hadoop.root.logger=WARN,console
>> > hadoop.root.logger=ERROR,console
>> >
>> > Depending on the logging information you require, of course.
>> >
>> > -Chuck
>> >
>> >
>> > -----Original Message-----
>> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
>> > Sent: Wednesday, February 13, 2013 11:32 AM
>> > To: [EMAIL PROTECTED]
>> > Subject: Managing space in Master Node
>> >
>> > Hello Gurus,
>> >
>> > I am managing a Hadoop Cluster to run some experiments.
>> >
>> > The issue I am continuously facing is that the Master Node runs out of
>> > disk
>> > space due to logs and data files.
>> >
>> > I can monitor and delete log files. However, I cannot delete the HDFS
>> > data.
>> >
>> > Thus, is there a way to force Hadoop not to save any HDFS data in the
>> > Master
>> > Node?
>> >
>> > Then I can use my master to handle the metadata only and store the logs.
>> >
>> > Thanks & regards
>> > Arko
>> > SDL Enterprise Technologies, Inc. - all rights reserved.  The
>> > information contained in this email may be confidential and/or legally
>> > privileged. It has been sent for the sole use of the intended recipient(s).
>> > If you are not the intended recipient of this mail, you are hereby notified
>> > that any unauthorized review, use, disclosure, dissemination, distribution,
>> > or copying of this communication, or any of its contents, is strictly
>> > prohibited. If you have received this communication in error, please reply
>> > to the sender and destroy all copies of the message.
>> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880,
>> > USA
>> >
>
>
+
Mohammad Tariq 2013-02-13, 20:06
+
Arko Provo Mukherjee 2013-02-13, 20:12
+
Charles Baker 2013-02-13, 20:21
+
Mohammad Tariq 2013-02-13, 20:14
+
Arko Provo Mukherjee 2013-02-13, 20:28
+
Arko Provo Mukherjee 2013-02-13, 20:31
+
Arko Provo Mukherjee 2013-02-13, 23:06
+
Mohammad Tariq 2013-02-13, 23:22
+
Arko Provo Mukherjee 2013-02-13, 23:32
+
Arko Provo Mukherjee 2013-02-13, 23:35
+
Mohammad Tariq 2013-02-13, 23:43
+
Arko Provo Mukherjee 2013-02-13, 23:48
+
Mohammad Tariq 2013-02-13, 23:51
+
Arko Provo Mukherjee 2013-02-13, 23:54
+
Mohammad Tariq 2013-02-14, 00:02
+
ramon.pin@... 2013-02-14, 09:31
+
Arko Provo Mukherjee 2013-02-14, 17:48
+
Arko Provo Mukherjee 2013-02-15, 00:21
+
Charles Baker 2013-02-15, 02:16