Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Managing space in Master Node


+
Arko Provo Mukherjee 2013-02-13, 19:32
+
Mohammad Tariq 2013-02-13, 19:37
+
Charles Baker 2013-02-13, 19:38
+
Arko Provo Mukherjee 2013-02-13, 19:52
+
Mohammad Tariq 2013-02-13, 19:55
+
Arko Provo Mukherjee 2013-02-13, 20:01
+
Mohammad Tariq 2013-02-13, 20:06
+
Arko Provo Mukherjee 2013-02-13, 20:12
+
Charles Baker 2013-02-13, 20:21
+
Mohammad Tariq 2013-02-13, 20:14
+
Arko Provo Mukherjee 2013-02-13, 20:28
Copy link to this message
-
Re: Managing space in Master Node
Arko Provo Mukherjee 2013-02-13, 20:31
Thanks! Let me study and find out if it works for me.
Thanks & regards
Arko

On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> But the new location will also be the same NN physical server.
> So how does that help?
>
> I would like to clear up the space in the master by removing all the
> HDFS data to the slave nodes.
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>> No, copy this directory to the new location of your choice.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>> <[EMAIL PROTECTED]> wrote:
>>>
>>> Thanks!
>>> However, in that case, how do I handle the current data that is already in
>>> NN?
>>> Do I have to reformat HDFS?
>>> Thanks & regards
>>> Arko
>>>
>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>> wrote:
>>> > Just don't run the DN daemon on that machine. Dedicate that machine to
>>> > NN.
>>> > Remove the hostname of the NN machine from the 'slaves' file.
>>> >
>>> > Warm Regards,
>>> > Tariq
>>> > https://mtariq.jux.com/
>>> > cloudfront.blogspot.com
>>> >
>>> >
>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>>> > <[EMAIL PROTECTED]> wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> Thanks for the help!
>>> >>
>>> >> However, I am still unsure about how to "turn off" the datanode
>>> >> feature on the NN.
>>> >>
>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>>> >> a directory.
>>> >>
>>> >> Should I just comment out that property? What would happen to the
>>> >> current data that there is in the Master? Will it get removed
>>> >> automatically?
>>> >>
>>> >> Thanks & regards
>>> >> Arko
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>> >> wrote:
>>> >> > You can specify the logging level as specified by Charles. But
>>> >> > turning
>>> >> > logs
>>> >> > off is never a good idea. Logs are really helpful in problem
>>> >> > diagnosis,
>>> >> > which are eventual.
>>> >> >
>>> >> > Warm Regards,
>>> >> > Tariq
>>> >> > https://mtariq.jux.com/
>>> >> > cloudfront.blogspot.com
>>> >> >
>>> >> >
>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>>> >> > <[EMAIL PROTECTED]> wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>>> >> >>
>>> >> >> I would like to "turn off" this feature.
>>> >> >>
>>> >> >> Request help regarding the same.
>>> >> >>
>>> >> >> Thanks & regards
>>> >> >> Arko
>>> >> >>
>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]>
>>> >> >> wrote:
>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
>>> >> >> > which
>>> >> >> > is
>>> >> >> > not recommended practice. Normally, the only files the NN stores
>>> >> >> > are
>>> >> >> > the
>>> >> >> > image and edits files. It does not store any actual HDFS data. If
>>> >> >> > you
>>> >> >> > must
>>> >> >> > run a DN on the NN, try turning down the logging in
>>> >> >> > /conf/log4j.properties:
>>> >> >> >
>>> >> >> > #hadoop.root.logger=INFO,console
>>> >> >> > #hadoop.root.logger=WARN,console
>>> >> >> > hadoop.root.logger=ERROR,console
>>> >> >> >
>>> >> >> > Depending on the logging information you require, of course.
>>> >> >> >
>>> >> >> > -Chuck
>>> >> >> >
>>> >> >> >
>>> >> >> > -----Original Message-----
>>> >> >> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]
>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>>> >> >> > To: [EMAIL PROTECTED]
>>> >> >> > Subject: Managing space in Master Node
>>> >> >> >
>>> >> >> > Hello Gurus,
>>> >> >> >
>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>>> >> >> >
>>> >> >> > The issue I am continuously facing is that the Master Node runs
>>> >> >> > out
>>> >> >> > of
>>> >> >> > disk
+
Arko Provo Mukherjee 2013-02-13, 23:06
+
Mohammad Tariq 2013-02-13, 23:22
+
Arko Provo Mukherjee 2013-02-13, 23:32
+
Arko Provo Mukherjee 2013-02-13, 23:35
+
Mohammad Tariq 2013-02-13, 23:43
+
Arko Provo Mukherjee 2013-02-13, 23:48
+
Mohammad Tariq 2013-02-13, 23:51
+
Arko Provo Mukherjee 2013-02-13, 23:54
+
Mohammad Tariq 2013-02-14, 00:02
+
ramon.pin@... 2013-02-14, 09:31
+
Arko Provo Mukherjee 2013-02-14, 17:48
+
Arko Provo Mukherjee 2013-02-15, 00:21
+
Charles Baker 2013-02-15, 02:16