Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Error: Flume to HDFS Sink


Copy link to this message
-
Error: Flume to HDFS Sink
I am getting a strange error when I am configuring Flume to write to a HDFS
Sink.

17 Jul 2013 20:36:46,469 ERROR [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
 - Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Selector channel not found: *NullChannel*
        at
org.apache.flume.channel.MultiplexingChannelSelector.getChannelListFromNames(MultiplexingChannelSelector.java:166)
Do I need to change my configure file, here is my flume-conf.properties

Second question, the hdfs.path - does it have to be hdfs namenode syntax

I am new to both flume and hadoop/hdfs.

# Name the components on this agent
agent1.sources = HTTPSource
agent1.sinks = PurePathSink UserActionSink VisitSink NullSink
agent1.channels = PurePathChannel UserActionChannel VisitChannel *NullChannel*

# Describe/configure HTTPSource
agent1.sources.HTTPSource.type = org.apache.flume.source.http.HTTPSource
agent1.sources.HTTPSource.port = 4321
agent1.sources.HTTPSource.handler com.dynatrace.diagnostics.btexport.flume.BtExportHandler

# Describe sinks
agent1.sinks.PurePathSink.type = hdfs
agent1.sinks.PurePathSink.hdfs.path = hdfs://localhost:9000/user/bts/pp
agent1.sinks.PurePathSink.hdfs.fileType = DataStream
agent1.sinks.PurePathSink.hdfs.filePrefix = export
agent1.sinks.PurePathSink.hdfs.fileSuffix = .txt
agent1.sinks.PurePathSink.hdfs.rollInterval = 120
agent1.sinks.PurePathSink.hdfs.rollSize = 131072
agent1.sinks.PurePathSink.serializer com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializerBuilder

agent1.sinks.UserActionSink.type = hdfs
agent1.sinks.UserActionSink.hdfs.path = hdfs://localhost:9000/user/bts/pa
agent1.sinks.UserActionSink.hdfs.fileType = DataStream
agent1.sinks.UserActionSink.hdfs.filePrefix = export
agent1.sinks.UserActionSink.hdfs.fileSuffix = .txt
agent1.sinks.UserActionSink.hdfs.rollInterval = 120
agent1.sinks.UserActionSink.hdfs.rollSize = 131072
agent1.sinks.UserActionSink.serializer com.dynatrace.diagnostics.btexport.flume.BtPageActionSerializerBuilder

agent1.sinks.VisitSink.type = hdfs
agent1.sinks.VisitSink.hdfs.path = hdfs://localhost:9000/user/bts/visit
agent1.sinks.VisitSink.hdfs.fileType = DataStream
agent1.sinks.VisitSink.hdfs.filePrefix = export
agent1.sinks.VisitSink.hdfs.fileSuffix = .txt
agent1.sinks.VisitSink.hdfs.rollInterval = 120
agent1.sinks.VisitSink.hdfs.rollSize = 131072
agent1.sinks.VisitSink.serializer com.dynatrace.diagnostics.btexport.flume.BtVisitSerializerBuilder

agent1.sinks.NullSink.type = null

# Use a channel which buffers events in memory
agent1.channels.PurePathChannel.type = memory
agent1.channels.PurePathChannel.capacity = 1000
agent1.channels.PurePathChannel.transactionCapactiy = 100

agent1.channels.UserActionChannel.type = memory
agent1.channels.UserActionChannel.capacity = 1000
agent1.channels.UserActionChannel.transactionCapactiy = 100

agent1.channels.VisitChannel.type = memory
agent1.channels.VisitChannel.capacity = 1000
agent1.channels.VisitChannel.transactionCapactiy = 100

# Bind the source and sink to the channel
agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel
VisitChannel NullChannel
agent1.sinks.PurePathSink.channel = PurePathChannel
agent1.sinks.UserActionSink.channel = UserActionChannel
agent1.sinks.VisitSink.channel = VisitChannel
agent1.sinks.NullSink.channel = NullChannel

agent1.sources.HTTPSource.selector.type = multiplexing
agent1.sources.HTTPSource.selector.header = btType
agent1.sources.HTTPSource.selector.mapping.PUREPATH = PurePathChannel
agent1.sources.HTTPSource.selector.mapping.PAGE_ACTION = UserActionChannel
agent1.sources.HTTPSource.selector.mapping.VISIT = VisitChannel
agent1.sources.HTTPSource.selector.default = NullChannel
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB