Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Re: Shutdown hook for FileSystems


Copy link to this message
-
Re: Shutdown hook for FileSystems
Oleg Zhurakousky 2014-01-22, 19:08
Sure it does. Your application most certainly has a starting point (some
Main), which starts-up everything else. So what you need to do is create
your own shutdown hook during the startup of the application so it could
catch Ctrl+C kind of termination and then within such shutdown hook
initiate a graceful shutdown.
Now I don't know how you create threads. I would certainly hope you are
using java.util.concurrent package and if so ExecutorService has few
shutdown(..) methods. This means you can call it inside of your Main's
shutdown hook. That of course would be a very, very bad idea since it would
require you to expose all executors to the Main. But Main obviously starts
something (a process) which means it has reference to it and such process
most definitely has reference to its supporting classes and so on. So what
I would recommend is for the Main's process to have a stop() method which
could be invoked in the Main's shutdown hook and such process can propagate
its own invocation of stop() to other "stoppable" components by invoking
their stop() methods and so on, thus achieving a graceful shutdown without
escaping your Executors, thus not breaking any encapsulation rules

Cheers
Oleg.
On Wed, Jan 22, 2014 at 12:37 PM, Lukas Kairies <
[EMAIL PROTECTED]> wrote:

>  Thanks for your answer, but I do not see how this could work in my case.
> My problem is that XtreemFS uses Threads (RPC clients) for method calls and
> therefore I have to stop this threads by myself to invoke the shutdown
> hooks. I think replacing one shutdown hook with another does not work in my
> case.My idea was to determine if the file system can be closed, because the
> mapper/reducer is done instead of waiting for the shutdown hook. So is this
> possible? E.g. getting the status of the task from somewhere an checking if
> it is finished?
>
> Am 22.01.2014 01:38, schrieb Oleg Zhurakousky:
>
> I am not sure either, you have to ask Hadoop guys, but it was giving me a
> hard time so I found a way around it.
>
>
> On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
>
>> I guess im not sure what the ShutdownHook actually is there for.... Thats
>> the real question im asking .
>>
>>
>> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
>> [EMAIL PROTECTED]> wrote:
>>
>>> No, all I do is have my own shutdown hook in the main which closes the
>>> FSDataOutputStream. Before I did that it would throw an ugly exception when
>>> I hit Ctrl+C, telling me that the stream is already closed, because of this
>>> shutdown hook (bad design on the hadoop part), so removing it keeps it open
>>> until the client's shutdown hook will close it at due time. So now it shuts
>>> down gracefully and under complete control of the client.
>>>
>>>
>>> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
>>>
>>>> what is happening when you remove the shutdown hook ?    is that
>>>> supposed to  trigger an exception -
>>>>
>>>
>>>
>>
>>
>>  --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>>
>
>
>