Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Re: [jira] [Created] (FLUME-1479) Multiple Sinks can connect to single Channel


Copy link to this message
-
Re: [jira] [Created] (FLUME-1479) Multiple Sinks can connect to single Channel
Hi Denny,

I am working on the patch now, it's not difficult. I have listed the
changes in that JIRA.
I think you misunderstand my design, I didn't maintain the order of the
events. Instead I make sure that each sink will get the same events (or
different events specified by selector).

Suppose Channel (mc) contains the following events: 4,3,2,1

If simply enable it by configuration, it may work like this:
Sink "hsa" may get 1,3;
Sink "hsb" may get 2,4;
So different sink will get different data. Is this what user wants?
In my design, "hsa" and "hsb" will both get "4,3,2,1". This is a typical
case when user want to fan-out the data into two places (eg. One for batch
and and another for real-time analysis).

Regards,
Yongkun Wang
On 12/08/10 14:29, "Denny Ye" <[EMAIL PROTECTED]> wrote:

>hi Yongkun,
>
>   JIRA can be accessed now.
>
>   I think it might be difficult to understand the order of events from
>your thought. If we don't care about the order, can discuss the value and
>feasibility.  In my opinion, data ingest flow is order unawareness, at
>least, not such important for us. You can try to verify your proposal and
>give us result. It may be some difficulties in keeping transaction with
>several Sinks.
>
>-Regards
>Denny Ye
>
>
>2012/8/10 Wang, Yongkun | Yongkun | BDD <[EMAIL PROTECTED]>
>
>> JIRA is down again? I cannot connect to it and comment there.
>>
>> I have a proposal in "Transactional Multiplex (fan out) Sink"):
>> https://issues.apache.org/jira/browse/FLUME-1435
>> Which contains the design of one channel to multiple sinks.
>>
>> You can search the email since JIRA cannot be accessed.
>>
>> I think this is more than a configuration issue. If simply enable
>>several
>> sinks on the same channel, they will take it either in a round-robin
>>mode
>> or in a unpredictable mode if the speed of sinks are different.
>>
>> So it's better to have a even higher level transaction control instead
>>of
>> the transaction in the process() of each sink, as I describe in
>>FLUME-1435.
>>
>> Regards,
>> Yongkun Wang
>>
>>
>> On 12/08/10 12:30, "Denny Ye (JIRA)" <[EMAIL PROTECTED]> wrote:
>>
>> >Denny Ye created FLUME-1479:
>> >-------------------------------
>> >
>> >             Summary: Multiple Sinks can connect to single Channel
>> >                 Key: FLUME-1479
>> >                 URL: https://issues.apache.org/jira/browse/FLUME-1479
>> >             Project: Flume
>> >          Issue Type: Bug
>> >          Components: Configuration
>> >    Affects Versions: v1.2.0
>> >            Reporter: Denny Ye
>> >            Assignee: Denny Ye
>> >             Fix For: v1.3.0
>> >
>> >
>> >If we has one Channel (mc) and two Sinks (hsa, hsb), then they may be
>> >connected with each other with configuration example
>> >{quote}
>> >agent.sinks.hsa.channel = mc
>> >agent.sinks.hsb.channel = mc
>> >{quote}
>> >It means that there have multiple Sinks can connect to single Channel.
>> >Normally, one Sink only can connect to unified Channel
>> >
>> >--
>> >This message is automatically generated by JIRA.
>> >If you think it was sent incorrectly, please contact your JIRA
>> >administrators:
>>
>>>https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
>> >For more information on JIRA, see:
>>http://www.atlassian.com/software/jira
>> >
>> >
>> >
>>
>>
>>