Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Restarts without data loss


Copy link to this message
-
Re: Restarts without data loss
Arvind Prabhakar 2012-07-10, 09:23
One clarification - as Mubarak mentioned, there is already a Jira for this
FLUME-1318 <https://issues.apache.org/jira/browse/FLUME-1318>. So instead
of filing a new issue, you can add your details and thoughts to this.

Regards,
Arvind Prabhakar

On Tue, Jul 10, 2012 at 2:20 AM, Arvind Prabhakar <[EMAIL PROTECTED]> wrote:

> Hi Inder,
> On Mon, Jul 9, 2012 at 12:02 AM, Inder Pall <[EMAIL PROTECTED]> wrote:
>
>> Arvind,
>>
>> to me this is an important use-case for frequent prod rollouts. How about
>> thinking in the direction of supporting graceful shutdown for agents.
>>
>
> I do believe that the Agent does shutdown gracefully on interrupt.
> Specifically the components are started in a specific order (FLUME-1236<https://issues.apache.org/jira/browse/FLUME-1236>)
> and then shutdown in the reverse order (FLUME-1325<https://issues.apache.org/jira/browse/FLUME-1325>).
> If you find that is not the case, do please file a Jira with appropriate
> details.
>
>
>> I can't think of an elegant solution at the moment which will address all
>> cases however what are thoughts regarding something on the lines ->
>>
>> 1. agent receives a shutdown signal.
>> 2. puts all it's channel in isolation mode(wherein no sources can put
>> stuff into it.)
>> 3. when the sinks attached to this channel drain( we do the real
>> shutdown).
>>
>> i know we can find issues with this algo however i want to highlight the
>> importance of graceful shutdown being supported as a first class use-case
>> here.
>>
>
> I guess what you are asking for is a drain-and-shutdown semantic. I think
> it is a perfectly reasonable request and something we should consider
> carefully as it will likely be used in production environments. In order to
> implement that, we would need to first create a system that allows the
> ability to send soft-interrupts such as commands via a socket and then
> create an implementation that provides for the semantics you describe
> above, along with regular shutdown semantics.
>
> The best bet to go about this would be start by filing a Jira, and adding
> as many details as you can to clearly specify it. And perhaps even taking a
> crack at doing a patch for it!
>
> Regards,
> Arvind Prabhakar
>
>
>>
>> - Inder
>>
>>
>> On Mon, Jul 9, 2012 at 12:16 PM, alo alt <[EMAIL PROTECTED]> wrote:
>>
>>> Simple solution:
>>>
>>> Two configs on different ports, iptables with transparent forwarding to
>>> both ports. Block the first one, all events will be redirected to the other
>>> port. Wait 5 minutes, the mem channel should be clear now. Do you changes,
>>> start the new config, redirect the traffic to these port and change the
>>> other config.
>>>
>>> cheers,
>>>  alex
>>>
>>>
>>> On Jul 9, 2012, at 8:29 AM, Arvind Prabhakar wrote:
>>>
>>> > Hi,
>>> >
>>> > On Sun, Jul 8, 2012 at 11:18 PM, Senthilvel Rangaswamy <
>>> [EMAIL PROTECTED]
>>> >> wrote:
>>> >
>>> >> We are using Flume 1.2.0 with memory channel. When we rollout new
>>> >> configs/decorators
>>> >> we may need to restart flume at which point any events in memory
>>> channel
>>> >> is gone. Any
>>> >> ways to avoid this ?
>>> >>
>>> >
>>> > One way to address this would be to make sure that the upstream sink or
>>> > client can be routed to a different agent when necessary. That way
>>> when you
>>> > do want to restart the file channel, you would first route all the
>>> traffic
>>> > elsewhere, drain the channel and then do the shutdown as necessary.
>>> Once
>>> > the system is back up, you could route the traffic back to this agent.
>>> >
>>> > I am sure that there are multiple other ways of doing this.
>>> >
>>> > Regards,
>>> > Arvind Prabhakar
>>> >
>>> >
>>> >
>>> >>
>>> >> Thanks,
>>> >> --
>>> >> ..Senthil
>>> >>
>>> >> "If there's anything more important than my ego around, I want it
>>> >> caught and shot now."
>>> >>                                                    - Douglas Adams.
>>> >>
>>> >>
>>>
>>>
>>> --
>>> Alexander Alten-Lorenz
>>> http://mapredit.blogspot.com