Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Multiplexing to multiple JdbcChannel (Derby) + event header ?


+
Alain B. 2013-01-22, 10:51
Copy link to this message
-
Re: Multiplexing to multiple JdbcChannel (Derby) + event header ?
I'd recommend using the FileChannel instead of the JDBC Channel. The
FileChannel would give you better performance and the same reliability
guarantees as the JDBC Channel.

To answer your question - you can insert custom headers in an agent using
Interceptors. You can write an interceptor which inserts the headers based
on the content of the event etc. Interceptors are available in version 1.2+.
Hari
On Tue, Jan 22, 2013 at 2:51 AM, Alain B. <[EMAIL PROTECTED]> wrote:

>
> Hi,
> I'd like to use the multiplexing feature of Flume-NG, routing events from
> one source to 2 jdbc-channels.
> My question is: will these 2 channels store their events in separate derby
> DB by default or do I need to configure my 2 jdbc-channels with specific
> properties in order to get 2 embedded derby DB started ?
> Moreover, the multiplexing feature can base its routing on headers. But no
> information is given on how to setup custom headers (the example in the
> documentation uses a "State" header).
> I'm using a Log4jAppender to write to an AvroSource.
>
> # list the sources, sinks and channels in the agent
> agent_foo.sources = avro-AppSrv-source1
> agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
> agent_foo.channels = jdbc-channel-1 jdbc-channel-2
>
> # set channels for source
> agent_foo.sources.avro-AppSrv-source1.channels = jdbc-channel-1
> jdbc-channel-2
>
> # set channel for sinks
> agent_foo.sinks.hdfs-Cluster1-sink1.channel = jdbc-channel-1
> agent_foo.sinks.avro-forward-sink2.channel = jdbc-channel-2
>
> # channel selector configuration
> agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
> agent_foo.sources.avro-AppSrv-source1.selector.header = State
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = jdbc-channel-1
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = jdbc-channel-2
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = jdbc-channel-1
> jdbc-channel-2
> agent_foo.sources.avro-AppSrv-source1.selector.default = jdbc-channel-1
>
> Thanks for your help,
> Best regards
+
Jeff Lord 2013-01-22, 18:54
+
Alain B. 2013-01-23, 08:01
+
Hari Shreedharan 2013-01-23, 08:05
+
Alain B. 2013-01-23, 08:07
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB