Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Managing space in Master Node


Copy link to this message
-
Re: Managing space in Master Node
Arko Provo Mukherjee 2013-02-13, 23:35
Hi,

I see that the NameNode IP is present in conf/slaves file.

Will removing the IP of the NN from the conf/slaves file and
restarting hadoop help?

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
> (running DN and Task Tracker).
>
> My issue is that the my NN is getting filled up pretty fast as the NN
> acts as a DN.
>
> I want my NN not to function as the DN.
>
> Does including the IP for the NN in the "dfs.hosts.exclude" and
> running "$ hadoop dfsadmin -refreshNodes" solve my problem?
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>> Stopping DN on one machine means excluding it. What exactly is the
>> requirement? How many nodes do you have right now?
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
>> <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi,
>>>
>>> I am not sure the provided link matches my situation.
>>>
>>> That is because, I am not trying to exclude any node but just trying
>>> to stop the NN act as a DataNode.
>>>
>>> Thanks & regards
>>> Arko
>>>
>>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
>>> <[EMAIL PROTECTED]> wrote:
>>> > Thanks! Let me study and find out if it works for me.
>>> > Thanks & regards
>>> > Arko
>>> >
>>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
>>> > <[EMAIL PROTECTED]> wrote:
>>> >> Hi,
>>> >>
>>> >> But the new location will also be the same NN physical server.
>>> >> So how does that help?
>>> >>
>>> >> I would like to clear up the space in the master by removing all the
>>> >> HDFS data to the slave nodes.
>>> >>
>>> >> Thanks & regards
>>> >> Arko
>>> >>
>>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>> >> wrote:
>>> >>> No, copy this directory to the new location of your choice.
>>> >>>
>>> >>> Warm Regards,
>>> >>> Tariq
>>> >>> https://mtariq.jux.com/
>>> >>> cloudfront.blogspot.com
>>> >>>
>>> >>>
>>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>>> >>> <[EMAIL PROTECTED]> wrote:
>>> >>>>
>>> >>>> Thanks!
>>> >>>> However, in that case, how do I handle the current data that is
>>> >>>> already in
>>> >>>> NN?
>>> >>>> Do I have to reformat HDFS?
>>> >>>> Thanks & regards
>>> >>>> Arko
>>> >>>>
>>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>>> >>>> wrote:
>>> >>>> > Just don't run the DN daemon on that machine. Dedicate that machine
>>> >>>> > to
>>> >>>> > NN.
>>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
>>> >>>> >
>>> >>>> > Warm Regards,
>>> >>>> > Tariq
>>> >>>> > https://mtariq.jux.com/
>>> >>>> > cloudfront.blogspot.com
>>> >>>> >
>>> >>>> >
>>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>>> >>>> > <[EMAIL PROTECTED]> wrote:
>>> >>>> >>
>>> >>>> >> Hi,
>>> >>>> >>
>>> >>>> >> Thanks for the help!
>>> >>>> >>
>>> >>>> >> However, I am still unsure about how to "turn off" the datanode
>>> >>>> >> feature on the NN.
>>> >>>> >>
>>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed
>>> >>>> >> to
>>> >>>> >> a directory.
>>> >>>> >>
>>> >>>> >> Should I just comment out that property? What would happen to the
>>> >>>> >> current data that there is in the Master? Will it get removed
>>> >>>> >> automatically?
>>> >>>> >>
>>> >>>> >> Thanks & regards
>>> >>>> >> Arko
>>> >>>> >>
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
>>> >>>> >> <[EMAIL PROTECTED]>
>>> >>>> >> wrote:
>>> >>>> >> > You can specify the logging level as specified by Charles. But
>>> >>>> >> > turning
>>> >>>> >> > logs
>>> >>>> >> > off is never a good idea. Logs are really helpful in problem
>>> >>>> >> > diagnosis,
>>> >>>> >> > which are eventual.
>>>