Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Restarts without data loss


Copy link to this message
-
Re: Restarts without data loss
Inder Pall 2012-07-10, 16:11
Thanks Arvind.

- Inder

On Tue, Jul 10, 2012 at 2:53 PM, Arvind Prabhakar <[EMAIL PROTECTED]> wrote:

> One clarification - as Mubarak mentioned, there is already a Jira for this
> FLUME-1318 <https://issues.apache.org/jira/browse/FLUME-1318>. So instead
> of filing a new issue, you can add your details and thoughts to this.
>
> Regards,
> Arvind Prabhakar
>
>
> On Tue, Jul 10, 2012 at 2:20 AM, Arvind Prabhakar <[EMAIL PROTECTED]>wrote:
>
>> Hi Inder,
>> On Mon, Jul 9, 2012 at 12:02 AM, Inder Pall <[EMAIL PROTECTED]> wrote:
>>
>>> Arvind,
>>>
>>> to me this is an important use-case for frequent prod rollouts. How
>>> about thinking in the direction of supporting graceful shutdown for agents.
>>>
>>
>> I do believe that the Agent does shutdown gracefully on interrupt.
>> Specifically the components are started in a specific order (FLUME-1236<https://issues.apache.org/jira/browse/FLUME-1236>)
>> and then shutdown in the reverse order (FLUME-1325<https://issues.apache.org/jira/browse/FLUME-1325>).
>> If you find that is not the case, do please file a Jira with appropriate
>> details.
>>
>>
>>> I can't think of an elegant solution at the moment which will address
>>> all cases however what are thoughts regarding something on the lines ->
>>>
>>> 1. agent receives a shutdown signal.
>>> 2. puts all it's channel in isolation mode(wherein no sources can put
>>> stuff into it.)
>>> 3. when the sinks attached to this channel drain( we do the real
>>> shutdown).
>>>
>>> i know we can find issues with this algo however i want to highlight the
>>> importance of graceful shutdown being supported as a first class use-case
>>> here.
>>>
>>
>> I guess what you are asking for is a drain-and-shutdown semantic. I think
>> it is a perfectly reasonable request and something we should consider
>> carefully as it will likely be used in production environments. In order to
>> implement that, we would need to first create a system that allows the
>> ability to send soft-interrupts such as commands via a socket and then
>> create an implementation that provides for the semantics you describe
>> above, along with regular shutdown semantics.
>>
>> The best bet to go about this would be start by filing a Jira, and adding
>> as many details as you can to clearly specify it. And perhaps even taking a
>> crack at doing a patch for it!
>>
>> Regards,
>> Arvind Prabhakar
>>
>>
>>>
>>> - Inder
>>>
>>>
>>> On Mon, Jul 9, 2012 at 12:16 PM, alo alt <[EMAIL PROTECTED]> wrote:
>>>
>>>> Simple solution:
>>>>
>>>> Two configs on different ports, iptables with transparent forwarding to
>>>> both ports. Block the first one, all events will be redirected to the other
>>>> port. Wait 5 minutes, the mem channel should be clear now. Do you changes,
>>>> start the new config, redirect the traffic to these port and change the
>>>> other config.
>>>>
>>>> cheers,
>>>>  alex
>>>>
>>>>
>>>> On Jul 9, 2012, at 8:29 AM, Arvind Prabhakar wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > On Sun, Jul 8, 2012 at 11:18 PM, Senthilvel Rangaswamy <
>>>> [EMAIL PROTECTED]
>>>> >> wrote:
>>>> >
>>>> >> We are using Flume 1.2.0 with memory channel. When we rollout new
>>>> >> configs/decorators
>>>> >> we may need to restart flume at which point any events in memory
>>>> channel
>>>> >> is gone. Any
>>>> >> ways to avoid this ?
>>>> >>
>>>> >
>>>> > One way to address this would be to make sure that the upstream sink
>>>> or
>>>> > client can be routed to a different agent when necessary. That way
>>>> when you
>>>> > do want to restart the file channel, you would first route all the
>>>> traffic
>>>> > elsewhere, drain the channel and then do the shutdown as necessary.
>>>> Once
>>>> > the system is back up, you could route the traffic back to this agent.
>>>> >
>>>> > I am sure that there are multiple other ways of doing this.
>>>> >
>>>> > Regards,
>>>> > Arvind Prabhakar
>>>> >
>>>> >
>>>> >
>>>> >>
>>>> >> Thanks,
>>>> >> --
>>>> >> ..Senthil
>>>> >>
>>>> >> "If there's anything more important than my ego around, I want it
Thanks,
- Inder
  Tech Platforms @Inmobi
  Linkedin - http://goo.gl/eR4Ub