Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Fwd: Issue with HBase Sink in Flume ( 1.3.0)


Copy link to this message
-
Fwd: Issue with HBase Sink in Flume ( 1.3.0)
Kris Ogirri 2014-02-17, 13:59
Dear Mailing Group,

I am currently having issues with the Hbase sink function. I have developed
an agent with a fanout channel setup ( single source, multiple channels,
multiple sinks) sinking to a HDFS cluster and Hbase deployment.

 The issue is that although the HDFS is working well, the Hbase flow is
simply not working. There are no errors being reported by Flume for the
Hbase channel but there are never any records being written to the HBase
store. The Hbase table as stipulated in the config always remains empty.
Studying the Flume startup logs I observe that the session connection to
Zookeeper is always successfully established

Are there any special configurations I am missing out?

I am using the Async Event Serializer to persist the txns.

Any assistance will be greatly appreciated.
Please see below for the flume configuration:

[biadmin@bivm bin]$ cat flume-conf.properties.bigdemo
agent.sources=exec-source
agent.sinks=hdfs-sink hbase-sink
agent.channels=ch1 ch2

agent.sources.exec-source.type=exec
agent.sources.exec-source.command=tail -F
/home/biadmin/bigdemo/data/rec_telco.cdr

agent.sinks.hdfs-sink.type=hdfs
agent.sinks.hdfs-sink.hdfs.path=hdfs://XXXX:9000/user/biadmin/bigdemo/
agent.sinks.hdfs-sink.hdfs.filePrefix=telco_cdr_rec
# File size to trigger roll, in bytes (0: never roll based on file size)
agent.sinks.hdfs-sink.hdfs.rollSize = 134217728
agent.sinks.hdfs-sink.hdfs.rollCount = 0
# number of events written to file before it flushed to HDFS
agent.sinks.hdfs-sink.hdfs.batchSize = 10000
agent.sinks.hdfs-sink.hdfs.txnEventMax = 40000
agent.sinks.hbase-sink.type=org.apache.flume.sink.hbase.AsyncHBaseSink
agent.sinks.hbase-sink.serializer=org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
agent.sinks.hbase-sink.table=telco_cdr_rec
agent.sinks.hbase-sink.columnFamily = colfam
agent.sinks.hbase-sink.channels = ch2
#agent.sinks.hbase-sink.hdfs.batchSize = 10000
#agent.sinks.hbase-sink.hdfs.txnEventMax = 40000
agent.channels.ch1.type=file
agent.channels.ch1.checkpointInterval=3000
agent.channels.ch1.transactionCapacity=10000
agent.channels.ch1.checkpointDir=/home/BDadmin/.flume/file-channel/checkpoint
agent.channels.ch1.dataDirs=/home/BDadmin/.flume/file-channel/data
agent.channels.ch1.write-timeout=30
agent.channels.ch1.keep-alive=30
#agent.channels.ch1.capacity=1000

agent.channels.ch2.type=file
agent.channels.ch2.checkpointInterval=300
agent.channels.ch2.transactionCapacity=10000
agent.channels.ch2.checkpointDir=/home/BDadmin/.flume/file-channel2/checkpoint
agent.channels.ch2.dataDirs=/home/BDadmin/.flume/file-channel2/data
agent.channels.ch2.write-timeout=30
agent.channels.ch2.keep-alive=30
#agent.channels.ch2.capacity=1000
agent.sources.exec-source.channels=ch1 ch2
agent.sinks.hdfs-sink.channel=ch1
agent.sinks.hbase-sink.channel=ch2