Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> EventDrivenSource and dead threads

Juhani Connolly 2013-01-17, 03:45
Brock Noland 2013-01-17, 04:20
Juhani Connolly 2013-01-17, 05:08
Connor Woodson 2013-01-17, 06:01
Juhani Connolly 2013-01-17, 06:21
Connor Woodson 2013-01-17, 07:02
Copy link to this message
Re: EventDrivenSource and dead threads

Yeah I'd prefer to avoid overriding getLifeCycle to do this since
getters are typically lightweight. As such, I feel it violates the
general contract behind getters.

In regards to channels/sinks/pollable sources, I don't think they need
this behavior as sinks and poll-able sources both have threads polling
them at regular intervals. We have two robust channels memory and file
which don't require this behavior as well. Given this, I feel like
this is limited to EventDrivenSources which don't have a thread
polling them.

On second thought, FLUME-1630 got rid of one layer of threads in the
flume agent, perhaps adding a single threaded scheduled executor to
EventDrivenSource is not a bad solution.


On Wed, Jan 16, 2013 at 11:02 PM, Connor Woodson <[EMAIL PROTECTED]> wrote:
> What I was trying to say is that the ScribeSource should not be responsible
> for restarting itself (just from my understanding of your idea; it breaks
> the existing paradigm as components should not have to control their own
> lifecycle). Going from Brock's link, I feel the most dynamic solution would
> be possible to just add in a lifecyle state RESTART and place it in that
> switch statement; when that state is reached, it tries to stop then restart
> the component and then sets the desired state to START (or STOP if it
> couldn't start it again, or set error=true if it couldn't stop it).
> And in a way to prevent overriding getLifecycleState to return RESTART,
> there could either be an inherited function from AbstractSource to call for
> a restart, there could maybe be an option to set it to RESTART on crash, or
> something else.
> I'll admit though that I know little about the lifecycle system, so I have
> no idea if this idea is any better.
> - Connor
> On Wed, Jan 16, 2013 at 10:21 PM, Juhani Connolly <
>> Sink, Source and Channel all extend LifecycleAware, so the function is
>> available to all components already.
>> I was more questioning whether it was reasonable to start including logic
>> to determine the state. That being said, I think the precedent of just
>> returning the state set by start/stop is more one of habit, so on further
>> thought I don't see it as being unreasonable. I'm going to give fixing
>> ScribeSource with it a poke.
>> As to new lifecycle states, I took a pass at reworking the lifecycle model
>> with guavas service implementation in the past, but it took some very
>> significant changes and didn't get the momentum/interest necessary to keep
>> working on it. That ticket is here https://issues.apache.org/**
>> jira/browse/FLUME-966 <https://issues.apache.org/jira/browse/FLUME-966> .
>> Brock might be working on it now though the issue doesn't appear to have
>> had attention since october.
>> On 01/17/2013 03:01 PM, Connor Woodson wrote:
>>> Why limit it to the sources? If there is going to be a change to one
>>> component's lifecycle, then I see no reason not to change every
>>> component's
>>> lifecycle.
>>> Sinks and Channels could very well have this problem; so what about giving
>>> each LifecycleAware component a takePulse method (or something); or to
>>> avoid creating a new method, add a new lifecycle state 'crashed' or such
>>> which, when detected, causes a restart of the component. Then components
>>> would just need to override getLifecycleState (if this method is polled
>>> regularly; I don't know. maybe use a listener for when there's a state
>>> change) to detect if it has crashed/needs to be restarted.
>>> Just my thoughts,
>>> - Connor
>>> On Wed, Jan 16, 2013 at 9:08 PM, Juhani Connolly <
>>> wrote:
>>>  Hmm, overriding the implementation of getLifecycleState provided by
>>>> AbstractSource could work. It would be going against the convention that
>>>> has been maintained in all other components(that I can think of)

Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Juhani Connolly 2013-01-17, 08:27
Connor Woodson 2013-01-17, 08:54