Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Multiplexing to multiple JdbcChannel (Derby) + event header ?


Copy link to this message
-
Multiplexing to multiple JdbcChannel (Derby) + event header ?
Alain B. 2013-01-22, 10:51

Hi,
I'd like to use the multiplexing feature of Flume-NG, routing events from one source to 2 jdbc-channels.
My question is: will these 2 channels store their events in separate derby DB by default or do I need to configure my 2 jdbc-channels with specific properties in order to get 2 embedded derby DB started ?
Moreover, the multiplexing feature can base its routing on headers. But no information is given on how to setup custom headers (the example in the documentation uses a "State" header).
I'm using a Log4jAppender to write to an AvroSource.

# list the sources, sinks and channels in the agent
agent_foo.sources = avro-AppSrv-source1
agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
agent_foo.channels = jdbc-channel-1 jdbc-channel-2

# set channels for source
agent_foo.sources.avro-AppSrv-source1.channels = jdbc-channel-1 jdbc-channel-2

# set channel for sinks
agent_foo.sinks.hdfs-Cluster1-sink1.channel = jdbc-channel-1
agent_foo.sinks.avro-forward-sink2.channel = jdbc-channel-2

# channel selector configuration
agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
agent_foo.sources.avro-AppSrv-source1.selector.header = State
agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = jdbc-channel-1
agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = jdbc-channel-2
agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = jdbc-channel-1 jdbc-channel-2
agent_foo.sources.avro-AppSrv-source1.selector.default = jdbc-channel-1

Thanks for your help,
Best regards