Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Managing space in Master Node


Copy link to this message
-
Re: Managing space in Master Node
Arko Provo Mukherjee 2013-02-13, 23:06
Hi,

I am not sure the provided link matches my situation.

That is because, I am not trying to exclude any node but just trying
to stop the NN act as a DataNode.

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
<[EMAIL PROTECTED]> wrote:
> Thanks! Let me study and find out if it works for me.
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> But the new location will also be the same NN physical server.
>> So how does that help?
>>
>> I would like to clear up the space in the master by removing all the
>> HDFS data to the slave nodes.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>>> No, copy this directory to the new location of your choice.
>>>
>>> Warm Regards,
>>> Tariq
>>> https://mtariq.jux.com/
>>> cloudfront.blogspot.com
>>>
>>>
>>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>>> <[EMAIL PROTECTED]> wrote:
>>>>
>>>> Thanks!
>>>> However, in that case, how do I handle the current data that is already in
>>>> NN?
>>>> Do I have to reformat HDFS?
>>>> Thanks & regards
>>>> Arko
>>>>
>>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>>> wrote:
>>>> > Just don't run the DN daemon on that machine. Dedicate that machine to
>>>> > NN.
>>>> > Remove the hostname of the NN machine from the 'slaves' file.
>>>> >
>>>> > Warm Regards,
>>>> > Tariq
>>>> > https://mtariq.jux.com/
>>>> > cloudfront.blogspot.com
>>>> >
>>>> >
>>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>>>> > <[EMAIL PROTECTED]> wrote:
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> Thanks for the help!
>>>> >>
>>>> >> However, I am still unsure about how to "turn off" the datanode
>>>> >> feature on the NN.
>>>> >>
>>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>>>> >> a directory.
>>>> >>
>>>> >> Should I just comment out that property? What would happen to the
>>>> >> current data that there is in the Master? Will it get removed
>>>> >> automatically?
>>>> >>
>>>> >> Thanks & regards
>>>> >> Arko
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>>> >> wrote:
>>>> >> > You can specify the logging level as specified by Charles. But
>>>> >> > turning
>>>> >> > logs
>>>> >> > off is never a good idea. Logs are really helpful in problem
>>>> >> > diagnosis,
>>>> >> > which are eventual.
>>>> >> >
>>>> >> > Warm Regards,
>>>> >> > Tariq
>>>> >> > https://mtariq.jux.com/
>>>> >> > cloudfront.blogspot.com
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>>>> >> > <[EMAIL PROTECTED]> wrote:
>>>> >> >>
>>>> >> >> Hi,
>>>> >> >>
>>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>>>> >> >>
>>>> >> >> I would like to "turn off" this feature.
>>>> >> >>
>>>> >> >> Request help regarding the same.
>>>> >> >>
>>>> >> >> Thanks & regards
>>>> >> >> Arko
>>>> >> >>
>>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <[EMAIL PROTECTED]>
>>>> >> >> wrote:
>>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
>>>> >> >> > which
>>>> >> >> > is
>>>> >> >> > not recommended practice. Normally, the only files the NN stores
>>>> >> >> > are
>>>> >> >> > the
>>>> >> >> > image and edits files. It does not store any actual HDFS data. If
>>>> >> >> > you
>>>> >> >> > must
>>>> >> >> > run a DN on the NN, try turning down the logging in
>>>> >> >> > /conf/log4j.properties:
>>>> >> >> >
>>>> >> >> > #hadoop.root.logger=INFO,console
>>>> >> >> > #hadoop.root.logger=WARN,console
>>>> >> >> > hadoop.root.logger=ERROR,console
>>>> >> >> >
>>>> >> >> > Depending on the logging information you require, of course.
>>>> >> >> >
>>>> >> >> > -Chuck
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > -----Original Message-----
>>>> >> >> > From: Arko Provo Mukherjee [mailto:[EMAIL PROTECTED]]