Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables


Copy link to this message
-
RE: question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables
There is a configuration error in your multiplexing channel selector section. You are referencing ‘server-agent.sources.avor-Src.’ and it should be ‘server-agent.sources.mySrc.’. Otherwise, the configuration looks good and should satisfy your requirements.

From: terrey shih [mailto:[EMAIL PROTECTED]]
Sent: Friday, August 08, 2014 1:20 PM
To: [EMAIL PROTECTED]
Subject: Re: question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables

Hi,

One more thing.  I would like to know if one serializer can be used for two table.

thanks,
-Gary
server-agent.sources = mySrc
server-agent.sinks = hbase-sink1 hbase-sink2
server-agent.channels = C1 C2

# Describe/configure the source
server-agent.sources.mySrc.type = avro
server-agent.sources.mySrc.bind = 127.0.0.1
server-agent.sources.mySrc.port = 5000

# Use a channel which buffers events in memory
server-agent.channels.C1.type = memory
server-agent.channels.C1.capacity = 1000
server-agent.channels.C1.transactionCapacity = 100
server-agent.sinks.hbase-sink1.type = asynchbase
server-agent.sinks.hbase-sink1.table = table1
server-agent.sinks.hbase-sink1.columnFamily = fam1
server-agent.sinks.hbase-sink1.batchSize = 1000
server-agent.sinks.hbase-sink1.serializer = com.test.flume.server.HBaseSinkSerializer
server-agent.sinks.hbase-sink1.channel = C1
server-agent.sinks.hbase-sink1.serializer.columns = col1,col2,col3

server-agent.channels.C2.type = memory
server-agent.channels.C2.capacity = 1000
server-agent.channels.C2.transactionCapacity = 100
server-agent.sinks.hbase-sink2.type = asynchbase
server-agent.sinks.hbase-sink2.table = table2
server-agent.sinks.hbase-sink2.columnFamily = fam2
server-agent.sinks.hbase-sink2.batchSize = 1000
server-agent.sinks.hbase-sink2.serializer = com.test.flume.server.HBaseSinkSerializer
server-agent.sinks.hbase-sink2.channel = C2
server-agent.sinks.hbase-sink2.serializer.columns = table2Col

# Bind the source and sink to the channel
server-agent.sources.mySrc.channels = C1 C2
server-agent.sources.avor-Src.selector.type = multiplexing
server-agent.sources.avor-Src.selector.header = DataSrc
server-agent.sources.avor-Src.selector.mapping.ABC = C1
server-agent.sources.avor-Src.selector.mapping.BCD = C2

On Fri, Aug 8, 2014 at 11:19 AM, terrey shih <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
HI,

Here is my sample config.

On Fri, Aug 8, 2014 at 10:52 AM, Hari Shreedharan <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Can you please send your config? That would make it easier to understand.
terrey shih wrote:
I have a fanning out operation where I have one source and and based
on the event headers (headers are added from the source input), I
would like to channel the event to different Hbase tables.  I am using
the same serializer for the hbase tables.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB