Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Managing space in Master Node


+
Arko Provo Mukherjee 2013-02-13, 19:32
+
Mohammad Tariq 2013-02-13, 19:37
+
Charles Baker 2013-02-13, 19:38
+
Arko Provo Mukherjee 2013-02-13, 19:52
+
Mohammad Tariq 2013-02-13, 19:55
Copy link to this message
-
Re: Managing space in Master Node
Hi,

Thanks for the help!

However, I am still unsure about how to "turn off" the datanode
feature on the NN.

I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
a directory.

Should I just comment out that property? What would happen to the
current data that there is in the Master? Will it get removed
automatically?

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> You can specify the logging level as specified by Charles. But turning logs
> off is never a good idea. Logs are really helpful in problem diagnosis,
> which are eventual.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> Yeah, my NameNode is also seconding as a DataNode.
>>
>> I would like to "turn off" this feature.
>>
>> Request help regarding the same.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]> wrote:
>> > Hi Arko. Sounds like you may be running a DataNode on the NameNode which
>> > is
>> > not recommended practice. Normally, the only files the NN stores are the
>> > image and edits files. It does not store any actual HDFS data. If you
>> > must
>> > run a DN on the NN, try turning down the logging in
>> > /conf/log4j.properties:
>> >
>> > #hadoop.root.logger=INFO,console
>> > #hadoop.root.logger=WARN,console
>> > hadoop.root.logger=ERROR,console
>> >
>> > Depending on the logging information you require, of course.
>> >
>> > -Chuck
>> >
>> >
>> > -----Original Message-----
>> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
>> > Sent: Wednesday, February 13, 2013 11:32 AM
>> > To: [EMAIL PROTECTED]
>> > Subject: Managing space in Master Node
>> >
>> > Hello Gurus,
>> >
>> > I am managing a Hadoop Cluster to run some experiments.
>> >
>> > The issue I am continuously facing is that the Master Node runs out of
>> > disk
>> > space due to logs and data files.
>> >
>> > I can monitor and delete log files. However, I cannot delete the HDFS
>> > data.
>> >
>> > Thus, is there a way to force Hadoop not to save any HDFS data in the
>> > Master
>> > Node?
>> >
>> > Then I can use my master to handle the metadata only and store the logs.
>> >
>> > Thanks & regards
>> > Arko
>> > SDL Enterprise Technologies, Inc. - all rights reserved.  The
>> > information contained in this email may be confidential and/or legally
>> > privileged. It has been sent for the sole use of the intended recipient(s).
>> > If you are not the intended recipient of this mail, you are hereby notified
>> > that any unauthorized review, use, disclosure, dissemination, distribution,
>> > or copying of this communication, or any of its contents, is strictly
>> > prohibited. If you have received this communication in error, please reply
>> > to the sender and destroy all copies of the message.
>> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880,
>> > USA
>> >
>
>
+
Mohammad Tariq 2013-02-13, 20:06
+
Arko Provo Mukherjee 2013-02-13, 20:12
+
Charles Baker 2013-02-13, 20:21
+
Mohammad Tariq 2013-02-13, 20:14
+
Arko Provo Mukherjee 2013-02-13, 20:28
+
Arko Provo Mukherjee 2013-02-13, 20:31
+
Arko Provo Mukherjee 2013-02-13, 23:06
+
Mohammad Tariq 2013-02-13, 23:22
+
Arko Provo Mukherjee 2013-02-13, 23:32
+
Arko Provo Mukherjee 2013-02-13, 23:35
+
Mohammad Tariq 2013-02-13, 23:43
+
Arko Provo Mukherjee 2013-02-13, 23:48
+
Mohammad Tariq 2013-02-13, 23:51
+
Arko Provo Mukherjee 2013-02-13, 23:54
+
Mohammad Tariq 2013-02-14, 00:02
+
ramon.pin@... 2013-02-14, 09:31
+
Arko Provo Mukherjee 2013-02-14, 17:48
+
Arko Provo Mukherjee 2013-02-15, 00:21
+
Charles Baker 2013-02-15, 02:16
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB