Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Managing space in Master Node


+
Arko Provo Mukherjee 2013-02-13, 19:32
+
Mohammad Tariq 2013-02-13, 19:37
+
Charles Baker 2013-02-13, 19:38
+
Arko Provo Mukherjee 2013-02-13, 19:52
+
Mohammad Tariq 2013-02-13, 19:55
+
Arko Provo Mukherjee 2013-02-13, 20:01
+
Mohammad Tariq 2013-02-13, 20:06
+
Arko Provo Mukherjee 2013-02-13, 20:12
+
Charles Baker 2013-02-13, 20:21
+
Mohammad Tariq 2013-02-13, 20:14
Copy link to this message
-
Re: Managing space in Master Node
Hi,

But the new location will also be the same NN physical server.
So how does that help?

I would like to clear up the space in the master by removing all the
HDFS data to the slave nodes.

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> No, copy this directory to the new location of your choice.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>>
>> Thanks!
>> However, in that case, how do I handle the current data that is already in
>> NN?
>> Do I have to reformat HDFS?
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> wrote:
>> > Just don't run the DN daemon on that machine. Dedicate that machine to
>> > NN.
>> > Remove the hostname of the NN machine from the 'slaves' file.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>> > <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Thanks for the help!
>> >>
>> >> However, I am still unsure about how to "turn off" the datanode
>> >> feature on the NN.
>> >>
>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>> >> a directory.
>> >>
>> >> Should I just comment out that property? What would happen to the
>> >> current data that there is in the Master? Will it get removed
>> >> automatically?
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >>
>> >>
>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > You can specify the logging level as specified by Charles. But
>> >> > turning
>> >> > logs
>> >> > off is never a good idea. Logs are really helpful in problem
>> >> > diagnosis,
>> >> > which are eventual.
>> >> >
>> >> > Warm Regards,
>> >> > Tariq
>> >> > https://mtariq.jux.com/
>> >> > cloudfront.blogspot.com
>> >> >
>> >> >
>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> >> > <[EMAIL PROTECTED]> wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>> >> >>
>> >> >> I would like to "turn off" this feature.
>> >> >>
>> >> >> Request help regarding the same.
>> >> >>
>> >> >> Thanks & regards
>> >> >> Arko
>> >> >>
>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]>
>> >> >> wrote:
>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
>> >> >> > which
>> >> >> > is
>> >> >> > not recommended practice. Normally, the only files the NN stores
>> >> >> > are
>> >> >> > the
>> >> >> > image and edits files. It does not store any actual HDFS data. If
>> >> >> > you
>> >> >> > must
>> >> >> > run a DN on the NN, try turning down the logging in
>> >> >> > /conf/log4j.properties:
>> >> >> >
>> >> >> > #hadoop.root.logger=INFO,console
>> >> >> > #hadoop.root.logger=WARN,console
>> >> >> > hadoop.root.logger=ERROR,console
>> >> >> >
>> >> >> > Depending on the logging information you require, of course.
>> >> >> >
>> >> >> > -Chuck
>> >> >> >
>> >> >> >
>> >> >> > -----Original Message-----
>> >> >> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>> >> >> > To: [EMAIL PROTECTED]
>> >> >> > Subject: Managing space in Master Node
>> >> >> >
>> >> >> > Hello Gurus,
>> >> >> >
>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>> >> >> >
>> >> >> > The issue I am continuously facing is that the Master Node runs
>> >> >> > out
>> >> >> > of
>> >> >> > disk
>> >> >> > space due to logs and data files.
>> >> >> >
>> >> >> > I can monitor and delete log files. However, I cannot delete the
>> >> >> > HDFS
>> >> >> > data.
>> >> >> >
>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS data in
>> >> >> > the
>> >> >> > Master
>> >> >> > Node?
>> >> >> >
+
Arko Provo Mukherjee 2013-02-13, 20:31
+
Arko Provo Mukherjee 2013-02-13, 23:06
+
Mohammad Tariq 2013-02-13, 23:22
+
Arko Provo Mukherjee 2013-02-13, 23:32
+
Arko Provo Mukherjee 2013-02-13, 23:35
+
Mohammad Tariq 2013-02-13, 23:43
+
Arko Provo Mukherjee 2013-02-13, 23:48
+
Mohammad Tariq 2013-02-13, 23:51
+
Arko Provo Mukherjee 2013-02-13, 23:54
+
Mohammad Tariq 2013-02-14, 00:02
+
ramon.pin@... 2013-02-14, 09:31
+
Arko Provo Mukherjee 2013-02-14, 17:48
+
Arko Provo Mukherjee 2013-02-15, 00:21
+
Charles Baker 2013-02-15, 02:16