Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables


Copy link to this message
-
RE: question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables
There is a configuration error in your multiplexing channel selector section. You are referencing ‘server-agent.sources.avor-Src.’ and it should be ‘server-agent.sources.mySrc.’. Otherwise, the configuration looks good and should satisfy your requirements.

From: terrey shih [mailto:[EMAIL PROTECTED]]
Sent: Friday, August 08, 2014 1:20 PM
To: [EMAIL PROTECTED]
Subject: Re: question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables

Hi,

One more thing.  I would like to know if one serializer can be used for two table.

thanks,
-Gary
server-agent.sources = mySrc
server-agent.sinks = hbase-sink1 hbase-sink2
server-agent.channels = C1 C2

# Describe/configure the source
server-agent.sources.mySrc.type = avro
server-agent.sources.mySrc.bind = 127.0.0.1
server-agent.sources.mySrc.port = 5000

# Use a channel which buffers events in memory
server-agent.channels.C1.type = memory
server-agent.channels.C1.capacity = 1000
server-agent.channels.C1.transactionCapacity = 100
server-agent.sinks.hbase-sink1.type = asynchbase
server-agent.sinks.hbase-sink1.table = table1
server-agent.sinks.hbase-sink1.columnFamily = fam1
server-agent.sinks.hbase-sink1.batchSize = 1000
server-agent.sinks.hbase-sink1.serializer = com.test.flume.server.HBaseSinkSerializer
server-agent.sinks.hbase-sink1.channel = C1
server-agent.sinks.hbase-sink1.serializer.columns = col1,col2,col3

server-agent.channels.C2.type = memory
server-agent.channels.C2.capacity = 1000
server-agent.channels.C2.transactionCapacity = 100
server-agent.sinks.hbase-sink2.type = asynchbase
server-agent.sinks.hbase-sink2.table = table2
server-agent.sinks.hbase-sink2.columnFamily = fam2
server-agent.sinks.hbase-sink2.batchSize = 1000
server-agent.sinks.hbase-sink2.serializer = com.test.flume.server.HBaseSinkSerializer
server-agent.sinks.hbase-sink2.channel = C2
server-agent.sinks.hbase-sink2.serializer.columns = table2Col

# Bind the source and sink to the channel
server-agent.sources.mySrc.channels = C1 C2
server-agent.sources.avor-Src.selector.type = multiplexing
server-agent.sources.avor-Src.selector.header = DataSrc
server-agent.sources.avor-Src.selector.mapping.ABC = C1
server-agent.sources.avor-Src.selector.mapping.BCD = C2

On Fri, Aug 8, 2014 at 11:19 AM, terrey shih <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
HI,

Here is my sample config.

On Fri, Aug 8, 2014 at 10:52 AM, Hari Shreedharan <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Can you please send your config? That would make it easier to understand.
terrey shih wrote:
I have a fanning out operation where I have one source and and based
on the event headers (headers are added from the source input), I
would like to channel the event to different Hbase tables.  I am using
the same serializer for the hbase tables.