Hi,
I'd like to use the multiplexing feature of Flume-NG, routing events from one source to 2 jdbc-channels.
My question is: will these 2 channels store their events in separate derby DB by default or do I need to configure my 2 jdbc-channels with specific properties in order to get 2 embedded derby DB started ?
Moreover, the multiplexing feature can base its routing on headers. But no information is given on how to setup custom headers (the example in the documentation uses a "State" header).
I'm using a Log4jAppender to write to an AvroSource.

# list the sources, sinks and channels in the agent
agent_foo.sources = avro-AppSrv-source1
agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
agent_foo.channels = jdbc-channel-1 jdbc-channel-2

# set channels for source
agent_foo.sources.avro-AppSrv-source1.channels = jdbc-channel-1 jdbc-channel-2

# set channel for sinks
agent_foo.sinks.hdfs-Cluster1-sink1.channel = jdbc-channel-1
agent_foo.sinks.avro-forward-sink2.channel = jdbc-channel-2

# channel selector configuration
agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
agent_foo.sources.avro-AppSrv-source1.selector.header = State
agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = jdbc-channel-1
agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = jdbc-channel-2
agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = jdbc-channel-1 jdbc-channel-2
agent_foo.sources.avro-AppSrv-source1.selector.default = jdbc-channel-1

Thanks for your help,
Best regards    

I'd recommend using the FileChannel instead of the JDBC Channel. The
FileChannel would give you better performance and the same reliability
guarantees as the JDBC Channel.

To answer your question - you can insert custom headers in an agent using
Interceptors. You can write an interceptor which inserts the headers based
on the content of the event etc. Interceptors are available in version 1.2+.
Hari
On Tue, Jan 22, 2013 at 2:51 AM, Alain B. <[EMAIL PROTECTED]> wrote:

>
> Hi,
> I'd like to use the multiplexing feature of Flume-NG, routing events from
> one source to 2 jdbc-channels.
> My question is: will these 2 channels store their events in separate derby
> DB by default or do I need to configure my 2 jdbc-channels with specific
> properties in order to get 2 embedded derby DB started ?
> Moreover, the multiplexing feature can base its routing on headers. But no
> information is given on how to setup custom headers (the example in the
> documentation uses a "State" header).
> I'm using a Log4jAppender to write to an AvroSource.
>
> # list the sources, sinks and channels in the agent
> agent_foo.sources = avro-AppSrv-source1
> agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
> agent_foo.channels = jdbc-channel-1 jdbc-channel-2
>
> # set channels for source
> agent_foo.sources.avro-AppSrv-source1.channels = jdbc-channel-1
> jdbc-channel-2
>
> # set channel for sinks
> agent_foo.sinks.hdfs-Cluster1-sink1.channel = jdbc-channel-1
> agent_foo.sinks.avro-forward-sink2.channel = jdbc-channel-2
>
> # channel selector configuration
> agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
> agent_foo.sources.avro-AppSrv-source1.selector.header = State
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = jdbc-channel-1
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = jdbc-channel-2
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = jdbc-channel-1
> jdbc-channel-2
> agent_foo.sources.avro-AppSrv-source1.selector.default = jdbc-channel-1
>
> Thanks for your help,
> Best regards

On Tue, Jan 22, 2013 at 2:51 AM, Alain B. <[EMAIL PROTECTED]> wrote:

> My question is: will these 2 channels store their events in separate derby
> DB by default or do I need to configure my 2 jdbc-channels with specific
> properties in order to get 2 embedded derby DB started ?
>
By default they will use the same derby DB.
You can specify the location with the following property.

a1.channels.c1.driver.url = jdbc:derby:/var/run/flume-ng/channel1/
.flume/jdbc-channel/db;create=true

a1.channels.c2.driver.url = jdbc:derby:/var/run/flume-ng/channel2/
.flume/jdbc-channel/db;create=true

As Hari already mentioned we Highly recommend the file channel for
durability over the jdbc channel.

Ok, I got it thank you. I'll follow your advice and go for file channels instead.

W.r.t. the interceptor, I'll implement  a custom org.apache.flume.interceptor class.
Just one more question: interceptors are configured in the source configuration. But are they executed before the events are inserted in the source or when they get out of the source (before going to the channel). If it's the later case, then I suppose it's done before any selector does its job (multiplexing on the basis of a header in my specific case).

So is it :
log4jappender -> interceptor -> source -> selector -> channel
or
log4jappender -> source -> interceptor -> selector -> channel

Thanks,
Alain
________________________________
> Date: Tue, 22 Jan 2013 10:54:49 -0800
> Subject: Re: Multiplexing to multiple JdbcChannel (Derby) + event header ?
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
>
> On Tue, Jan 22, 2013 at 2:51 AM, Alain B.
> <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> My question is: will these 2 channels store their events in separate
> derby DB by default or do I need to configure my 2 jdbc-channels with
> specific properties in order to get 2 embedded derby DB started ?
>
>
> By default they will use the same derby DB.
> You can specify the location with the following property.
>
> a1.channels.c1.driver.url =
> jdbc:derby:/var/run/flume-ng/channel1/.flume/jdbc-channel/db;create=true
>
> a1.channels.c2.driver.url =
> jdbc:derby:/var/run/flume-ng/channel2/.flume/jdbc-channel/db;create=true
>
> As Hari already mentioned we Highly recommend the file channel for
> durability over the jdbc channel.
     

Alan,  

This is the path your event would follow:
log4jappender -> source -> interceptor -> selector (technically this is the channel processor, selector simply selects the channels) -> channel
Hari

--
Hari Shreedharan
On Wednesday, January 23, 2013 at 12:01 AM, Alain B. wrote:

> log4jappender -> source -> interceptor -> selector -> channel

Great ! I'll try and implement all this.

Thank you for your quick responses.

Best regards,
Alain

________________________________
> Date: Wed, 23 Jan 2013 00:05:56 -0800
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
> Subject: Re: Multiplexing to multiple JdbcChannel (Derby) + event header ?
>
> Alan,
>
> This is the path your event would follow:
> log4jappender -> source -> interceptor -> selector (technically this is
> the channel processor, selector simply selects the channels) -> channel
>
>
> Hari
>
> --
> Hari Shreedharan
>
>
> On Wednesday, January 23, 2013 at 12:01 AM, Alain B. wrote:
>
> log4jappender -> source -> interceptor -> selector -> channel
>