Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> help configuring flume scribe sink


Copy link to this message
-
help configuring flume scribe sink
I am working on a POC in which I am trying to have flume emulate scribe output by using a flume scribe sink that I found online.  There was no docs with it so I have poked through code to try and figure out how to configure this properly.  I have gotten the it configured to the point where a scribe server receives the data but does not recognize any scribe categories.  I have included my config file below.  In this config file I'm simply doing a tail -f on some bogus log files that I generate for the input and trying to create 3 streams of output similar to what a scribe intake server might be doing in receiving multiple streams of scribe data and funneling it to a scribe integration server.

Has anyone used this flume scribe sink successfully and, if so, can you help me get my config file setup properly?  Thank you.

Regards,
Dwight Marzolf

# Name the components on this agent
agent1.sources = exec-tail-ds1 exec-tail-ds2 exec-tail-ds3
agent1.sinks = scribe-forward-sink1 scribe-forward-sink2 scribe-forward-sink3
agent1.channels = file-channel-1

##########################
# SOURCE
##########################

# Configure source 1
agent1.sources.exec-tail-ds1.type = exec
agent1.sources.exec-tail-ds1.command = tail -F /tmp/dataset1
agent1.sources.exec-tail-ds1.channels = file-channel-1

# Configure source 2
agent1.sources.exec-tail-ds2.type = exec
agent1.sources.exec-tail-ds2.command = tail -F /tmp/dataset2
agent1.sources.exec-tail-ds2.channels = file-channel-1

# Configure source 3
agent1.sources.exec-tail-ds3.type = exec
agent1.sources.exec-tail-ds3.command = tail -F /tmp/dataset3
agent1.sources.exec-tail-ds3.channels = file-channel-1

##########################
# SINK
##########################

# Config for scribe sink 1
agent1.sinks.scribe-forward-sink1.type = org.apache.flume.sink.scribe.ScribeSink
agent1.sinks.scribe-forward-sink1.scribe.sink.name = ds1
agent1.sinks.scribe-forward-sink1.scribe.port = 1463
agent1.sinks.scribe-forward-sink1.scribe.host = 10.48.140.8
agent1.sinks.scribe-forward-sink1.scribe.category = dataset1
agent1.sinks.scribe-forward-sink1.scribe.category.header = category
agent1.sinks.scribe-forward-sink1.channel = file-channel-1

# Config for scribe sink 2
agent1.sinks.scribe-forward-sink2.type = org.apache.flume.sink.scribe.ScribeSink
agent1.sinks.scribe-forward-sink2.scribe.sink.name = ds2
agent1.sinks.scribe-forward-sink2.scribe.port = 1463
agent1.sinks.scribe-forward-sink2.scribe.host = 10.48.140.8
agent1.sinks.scribe-forward-sink2.scribe.category = dataset2
agent1.sinks.scribe-forward-sink2.scribe.category.header = category
agent1.sinks.scribe-forward-sink2.channel = file-channel-1

# Config for scribe sink 3
agent1.sinks.scribe-forward-sink3.type = org.apache.flume.sink.scribe.ScribeSink
agent1.sinks.scribe-forward-sink3.scribe.sink.name = ds3
agent1.sinks.scribe-forward-sink3.scribe.port = 1463
agent1.sinks.scribe-forward-sink3.scribe.host = 10.48.140.8
agent1.sinks.scribe-forward-sink3.scribe.category.header= category
agent1.sinks.scribe-forward-sink3.scribe.category = dataset3
agent1.sinks.scribe-forward-sink3.channel = file-channel-1

##########################
# CHANNEL
##########################

# Channel file buffer 1
agent1.channels.file-channel-1.type = file
agent1.channels.file-channel-1.checkpointDir = /scribe/channel/checkpoint1
agent1.channels.file-channel-1.dataDirs = /scribe/channel/data1

##########################
# BINDING
##########################

# Bind the source and sink to the channel
agent1.sources.exec-tail-ds1.channels = file-channel-1
agent1.sinks.scribe-forward-sink1.channel = file-channel-1
[Description: cid:B446C73C-E88B-4947-BD2A-586ECE21AB0D]
Dwight Marzolf
Sr. Engineer
Data & Analytics
Phone: +1 781 430 5228

HERE Burlington
5 Wayside Rd
42° 29' 7" N. 71° 11' 32" W
HERE, a Nokia business

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB