Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Managing space in Master Node


Copy link to this message
-
Re: Managing space in Master Node
Hi,

I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
(running DN and Task Tracker).

My issue is that the my NN is getting filled up pretty fast as the NN
acts as a DN.

I want my NN not to function as the DN.

Does including the IP for the NN in the "dfs.hosts.exclude" and
running "$ hadoop dfsadmin -refreshNodes" solve my problem?

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Stopping DN on one machine means excluding it. What exactly is the
> requirement? How many nodes do you have right now?
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> I am not sure the provided link matches my situation.
>>
>> That is because, I am not trying to exclude any node but just trying
>> to stop the NN act as a DataNode.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
>> <[EMAIL PROTECTED]> wrote:
>> > Thanks! Let me study and find out if it works for me.
>> > Thanks & regards
>> > Arko
>> >
>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
>> > <[EMAIL PROTECTED]> wrote:
>> >> Hi,
>> >>
>> >> But the new location will also be the same NN physical server.
>> >> So how does that help?
>> >>
>> >> I would like to clear up the space in the master by removing all the
>> >> HDFS data to the slave nodes.
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> >> wrote:
>> >>> No, copy this directory to the new location of your choice.
>> >>>
>> >>> Warm Regards,
>> >>> Tariq
>> >>> https://mtariq.jux.com/
>> >>> cloudfront.blogspot.com
>> >>>
>> >>>
>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>> >>> <[EMAIL PROTECTED]> wrote:
>> >>>>
>> >>>> Thanks!
>> >>>> However, in that case, how do I handle the current data that is
>> >>>> already in
>> >>>> NN?
>> >>>> Do I have to reformat HDFS?
>> >>>> Thanks & regards
>> >>>> Arko
>> >>>>
>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> >>>> wrote:
>> >>>> > Just don't run the DN daemon on that machine. Dedicate that machine
>> >>>> > to
>> >>>> > NN.
>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
>> >>>> >
>> >>>> > Warm Regards,
>> >>>> > Tariq
>> >>>> > https://mtariq.jux.com/
>> >>>> > cloudfront.blogspot.com
>> >>>> >
>> >>>> >
>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>> >>>> > <[EMAIL PROTECTED]> wrote:
>> >>>> >>
>> >>>> >> Hi,
>> >>>> >>
>> >>>> >> Thanks for the help!
>> >>>> >>
>> >>>> >> However, I am still unsure about how to "turn off" the datanode
>> >>>> >> feature on the NN.
>> >>>> >>
>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed
>> >>>> >> to
>> >>>> >> a directory.
>> >>>> >>
>> >>>> >> Should I just comment out that property? What would happen to the
>> >>>> >> current data that there is in the Master? Will it get removed
>> >>>> >> automatically?
>> >>>> >>
>> >>>> >> Thanks & regards
>> >>>> >> Arko
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
>> >>>> >> <[EMAIL PROTECTED]>
>> >>>> >> wrote:
>> >>>> >> > You can specify the logging level as specified by Charles. But
>> >>>> >> > turning
>> >>>> >> > logs
>> >>>> >> > off is never a good idea. Logs are really helpful in problem
>> >>>> >> > diagnosis,
>> >>>> >> > which are eventual.
>> >>>> >> >
>> >>>> >> > Warm Regards,
>> >>>> >> > Tariq
>> >>>> >> > https://mtariq.jux.com/
>> >>>> >> > cloudfront.blogspot.com
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> >>>> >> > <[EMAIL PROTECTED]> wrote:
>> >>>> >> >>
>> >>>> >> >> Hi,
>> >>>> >> >>
>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>> >>>> >> >>
>>