Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Error: Flume to HDFS Sink


Copy link to this message
-
Error: Flume to HDFS Sink
Rajesh Jain 2013-07-17, 20:47
I am getting a strange error when I am configuring Flume to write to a HDFS
Sink.

17 Jul 2013 20:36:46,469 ERROR [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
 - Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Selector channel not found: *NullChannel*
        at
org.apache.flume.channel.MultiplexingChannelSelector.getChannelListFromNames(MultiplexingChannelSelector.java:166)
Do I need to change my configure file, here is my flume-conf.properties

Second question, the hdfs.path - does it have to be hdfs namenode syntax

I am new to both flume and hadoop/hdfs.

# Name the components on this agent
agent1.sources = HTTPSource
agent1.sinks = PurePathSink UserActionSink VisitSink NullSink
agent1.channels = PurePathChannel UserActionChannel VisitChannel *NullChannel*

# Describe/configure HTTPSource
agent1.sources.HTTPSource.type = org.apache.flume.source.http.HTTPSource
agent1.sources.HTTPSource.port = 4321
agent1.sources.HTTPSource.handler com.dynatrace.diagnostics.btexport.flume.BtExportHandler

# Describe sinks
agent1.sinks.PurePathSink.type = hdfs
agent1.sinks.PurePathSink.hdfs.path = hdfs://localhost:9000/user/bts/pp
agent1.sinks.PurePathSink.hdfs.fileType = DataStream
agent1.sinks.PurePathSink.hdfs.filePrefix = export
agent1.sinks.PurePathSink.hdfs.fileSuffix = .txt
agent1.sinks.PurePathSink.hdfs.rollInterval = 120
agent1.sinks.PurePathSink.hdfs.rollSize = 131072
agent1.sinks.PurePathSink.serializer com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializerBuilder

agent1.sinks.UserActionSink.type = hdfs
agent1.sinks.UserActionSink.hdfs.path = hdfs://localhost:9000/user/bts/pa
agent1.sinks.UserActionSink.hdfs.fileType = DataStream
agent1.sinks.UserActionSink.hdfs.filePrefix = export
agent1.sinks.UserActionSink.hdfs.fileSuffix = .txt
agent1.sinks.UserActionSink.hdfs.rollInterval = 120
agent1.sinks.UserActionSink.hdfs.rollSize = 131072
agent1.sinks.UserActionSink.serializer com.dynatrace.diagnostics.btexport.flume.BtPageActionSerializerBuilder

agent1.sinks.VisitSink.type = hdfs
agent1.sinks.VisitSink.hdfs.path = hdfs://localhost:9000/user/bts/visit
agent1.sinks.VisitSink.hdfs.fileType = DataStream
agent1.sinks.VisitSink.hdfs.filePrefix = export
agent1.sinks.VisitSink.hdfs.fileSuffix = .txt
agent1.sinks.VisitSink.hdfs.rollInterval = 120
agent1.sinks.VisitSink.hdfs.rollSize = 131072
agent1.sinks.VisitSink.serializer com.dynatrace.diagnostics.btexport.flume.BtVisitSerializerBuilder

agent1.sinks.NullSink.type = null

# Use a channel which buffers events in memory
agent1.channels.PurePathChannel.type = memory
agent1.channels.PurePathChannel.capacity = 1000
agent1.channels.PurePathChannel.transactionCapactiy = 100

agent1.channels.UserActionChannel.type = memory
agent1.channels.UserActionChannel.capacity = 1000
agent1.channels.UserActionChannel.transactionCapactiy = 100

agent1.channels.VisitChannel.type = memory
agent1.channels.VisitChannel.capacity = 1000
agent1.channels.VisitChannel.transactionCapactiy = 100

# Bind the source and sink to the channel
agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel
VisitChannel NullChannel
agent1.sinks.PurePathSink.channel = PurePathChannel
agent1.sinks.UserActionSink.channel = UserActionChannel
agent1.sinks.VisitSink.channel = VisitChannel
agent1.sinks.NullSink.channel = NullChannel

agent1.sources.HTTPSource.selector.type = multiplexing
agent1.sources.HTTPSource.selector.header = btType
agent1.sources.HTTPSource.selector.mapping.PUREPATH = PurePathChannel
agent1.sources.HTTPSource.selector.mapping.PAGE_ACTION = UserActionChannel
agent1.sources.HTTPSource.selector.mapping.VISIT = VisitChannel
agent1.sources.HTTPSource.selector.default = NullChannel
+
Paul Chavez 2013-07-17, 21:03
+
Rajesh Jain 2013-07-17, 22:15