Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Multiplexing to multiple JdbcChannel (Derby) + event header ?


Copy link to this message
-
Multiplexing to multiple JdbcChannel (Derby) + event header ?

Hi,
I'd like to use the multiplexing feature of Flume-NG, routing events from one source to 2 jdbc-channels.
My question is: will these 2 channels store their events in separate derby DB by default or do I need to configure my 2 jdbc-channels with specific properties in order to get 2 embedded derby DB started ?
Moreover, the multiplexing feature can base its routing on headers. But no information is given on how to setup custom headers (the example in the documentation uses a "State" header).
I'm using a Log4jAppender to write to an AvroSource.

# list the sources, sinks and channels in the agent
agent_foo.sources = avro-AppSrv-source1
agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
agent_foo.channels = jdbc-channel-1 jdbc-channel-2

# set channels for source
agent_foo.sources.avro-AppSrv-source1.channels = jdbc-channel-1 jdbc-channel-2

# set channel for sinks
agent_foo.sinks.hdfs-Cluster1-sink1.channel = jdbc-channel-1
agent_foo.sinks.avro-forward-sink2.channel = jdbc-channel-2

# channel selector configuration
agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
agent_foo.sources.avro-AppSrv-source1.selector.header = State
agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = jdbc-channel-1
agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = jdbc-channel-2
agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = jdbc-channel-1 jdbc-channel-2
agent_foo.sources.avro-AppSrv-source1.selector.default = jdbc-channel-1

Thanks for your help,
Best regards    
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB