Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Error: Flume to HDFS Sink


Copy link to this message
-
RE: Error: Flume to HDFS Sink
You've specified a channel named NullChannel in the agent1.channels properties, but you don't define it anywhere in the configuration. You've done the same with the NullSink.

You'll need to add something like this to the config:

agent1.channels.NullChannel.type = memory

agent1.sinks.NullSink.type = null

Hope that helps,
Paul
________________________________
From: Rajesh Jain [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, July 17, 2013 1:47 PM
To: [EMAIL PROTECTED]
Subject: Error: Flume to HDFS Sink

I am getting a strange error when I am configuring Flume to write to a HDFS Sink.

17 Jul 2013 20:36:46,469 ERROR [conf-file-poller-0] (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)  - Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Selector channel not found: NullChannel
        at org.apache.flume.channel.MultiplexingChannelSelector.getChannelListFromNames(MultiplexingChannelSelector.java:166)
Do I need to change my configure file, here is my flume-conf.properties

Second question, the hdfs.path - does it have to be hdfs namenode syntax

I am new to both flume and hadoop/hdfs.
# Name the components on this agent
agent1.sources = HTTPSource
agent1.sinks = PurePathSink UserActionSink VisitSink NullSink
agent1.channels = PurePathChannel UserActionChannel VisitChannel NullChannel

# Describe/configure HTTPSource
agent1.sources.HTTPSource.type = org.apache.flume.source.http.HTTPSource
agent1.sources.HTTPSource.port = 4321
agent1.sources.HTTPSource.handler = com.dynatrace.diagnostics.btexport.flume.BtExportHandler

# Describe sinks
agent1.sinks.PurePathSink.type = hdfs
agent1.sinks.PurePathSink.hdfs.path = hdfs://localhost:9000/user/bts/pp
agent1.sinks.PurePathSink.hdfs.fileType = DataStream
agent1.sinks.PurePathSink.hdfs.filePrefix = export
agent1.sinks.PurePathSink.hdfs.fileSuffix = .txt
agent1.sinks.PurePathSink.hdfs.rollInterval = 120
agent1.sinks.PurePathSink.hdfs.rollSize = 131072
agent1.sinks.PurePathSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializerBuilder

agent1.sinks.UserActionSink.type = hdfs
agent1.sinks.UserActionSink.hdfs.path = hdfs://localhost:9000/user/bts/pa
agent1.sinks.UserActionSink.hdfs.fileType = DataStream
agent1.sinks.UserActionSink.hdfs.filePrefix = export
agent1.sinks.UserActionSink.hdfs.fileSuffix = .txt
agent1.sinks.UserActionSink.hdfs.rollInterval = 120
agent1.sinks.UserActionSink.hdfs.rollSize = 131072
agent1.sinks.UserActionSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPageActionSerializerBuilder

agent1.sinks.VisitSink.type = hdfs
agent1.sinks.VisitSink.hdfs.path = hdfs://localhost:9000/user/bts/visit
agent1.sinks.VisitSink.hdfs.fileType = DataStream
agent1.sinks.VisitSink.hdfs.filePrefix = export
agent1.sinks.VisitSink.hdfs.fileSuffix = .txt
agent1.sinks.VisitSink.hdfs.rollInterval = 120
agent1.sinks.VisitSink.hdfs.rollSize = 131072
agent1.sinks.VisitSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtVisitSerializerBuilder

agent1.sinks.NullSink.type = null

# Use a channel which buffers events in memory
agent1.channels.PurePathChannel.type = memory
agent1.channels.PurePathChannel.capacity = 1000
agent1.channels.PurePathChannel.transactionCapactiy = 100

agent1.channels.UserActionChannel.type = memory
agent1.channels.UserActionChannel.capacity = 1000
agent1.channels.UserActionChannel.transactionCapactiy = 100

agent1.channels.VisitChannel.type = memory
agent1.channels.VisitChannel.capacity = 1000
agent1.channels.VisitChannel.transactionCapactiy = 100

# Bind the source and sink to the channel
agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel VisitChannel NullChannel
agent1.sinks.PurePathSink.channel = PurePathChannel
agent1.sinks.UserActionSink.channel = UserActionChannel
agent1.sinks.VisitSink.channel = VisitChannel
agent1.sinks.NullSink.channel = NullChannel

agent1.sources.HTTPSource.selector.type = multiplexing
agent1.sources.HTTPSource.selector.header = btType
agent1.sources.HTTPSource.selector.mapping.PUREPATH = PurePathChannel
agent1.sources.HTTPSource.selector.mapping.PAGE_ACTION = UserActionChannel
agent1.sources.HTTPSource.selector.mapping.VISIT = VisitChannel
agent1.sources.HTTPSource.selector.default = NullChannel