Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Error: Flume to HDFS Sink


Copy link to this message
-
Re: Error: Flume to HDFS Sink
hi Paul

This did the trick.
agent1.channels.NullChannel.type = memory

Thanks
On Wed, Jul 17, 2013 at 5:03 PM, Paul Chavez <
[EMAIL PROTECTED]> wrote:

> **
> You've specified a channel named NullChannel in the agent1.channels
> properties, but you don't define it anywhere in the configuration. You've
> done the same with the NullSink.
>
> You'll need to add something like this to the config:
>
> agent1.channels.NullChannel.type = memory
>
> agent1.sinks.NullSink.type = null
>
> Hope that helps,
> Paul
>
>
>  ------------------------------
> *From:* Rajesh Jain [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, July 17, 2013 1:47 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Error: Flume to HDFS Sink
>
>  I am getting a strange error when I am configuring Flume to write to a
> HDFS Sink.
>
>  17 Jul 2013 20:36:46,469 ERROR [conf-file-poller-0]
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
>  - Failed to load configuration data. Exception follows.
> org.apache.flume.FlumeException: Selector channel not found: *NullChannel*
>         at
> org.apache.flume.channel.MultiplexingChannelSelector.getChannelListFromNames(MultiplexingChannelSelector.java:166)
>
>
> Do I need to change my configure file, here is my flume-conf.properties
>
> Second question, the hdfs.path - does it have to be hdfs namenode syntax
>
> I am new to both flume and hadoop/hdfs.
>
>
>
> # Name the components on this agent
> agent1.sources = HTTPSource
> agent1.sinks = PurePathSink UserActionSink VisitSink NullSink
> agent1.channels = PurePathChannel UserActionChannel VisitChannel *NullChannel*
>
> # Describe/configure HTTPSource
> agent1.sources.HTTPSource.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.HTTPSource.port = 4321
> agent1.sources.HTTPSource.handler = com.dynatrace.diagnostics.btexport.flume.BtExportHandler
>
> # Describe sinks
> agent1.sinks.PurePathSink.type = hdfs
> agent1.sinks.PurePathSink.hdfs.path = hdfs://localhost:9000/user/bts/pp
> agent1.sinks.PurePathSink.hdfs.fileType = DataStream
> agent1.sinks.PurePathSink.hdfs.filePrefix = export
> agent1.sinks.PurePathSink.hdfs.fileSuffix = .txt
> agent1.sinks.PurePathSink.hdfs.rollInterval = 120
> agent1.sinks.PurePathSink.hdfs.rollSize = 131072
> agent1.sinks.PurePathSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializerBuilder
>
> agent1.sinks.UserActionSink.type = hdfs
> agent1.sinks.UserActionSink.hdfs.path = hdfs://localhost:9000/user/bts/pa
> agent1.sinks.UserActionSink.hdfs.fileType = DataStream
> agent1.sinks.UserActionSink.hdfs.filePrefix = export
> agent1.sinks.UserActionSink.hdfs.fileSuffix = .txt
> agent1.sinks.UserActionSink.hdfs.rollInterval = 120
> agent1.sinks.UserActionSink.hdfs.rollSize = 131072
> agent1.sinks.UserActionSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPageActionSerializerBuilder
>
> agent1.sinks.VisitSink.type = hdfs
> agent1.sinks.VisitSink.hdfs.path = hdfs://localhost:9000/user/bts/visit
> agent1.sinks.VisitSink.hdfs.fileType = DataStream
> agent1.sinks.VisitSink.hdfs.filePrefix = export
> agent1.sinks.VisitSink.hdfs.fileSuffix = .txt
> agent1.sinks.VisitSink.hdfs.rollInterval = 120
> agent1.sinks.VisitSink.hdfs.rollSize = 131072
> agent1.sinks.VisitSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtVisitSerializerBuilder
>
> agent1.sinks.NullSink.type = null
>
> # Use a channel which buffers events in memory
> agent1.channels.PurePathChannel.type = memory
> agent1.channels.PurePathChannel.capacity = 1000
> agent1.channels.PurePathChannel.transactionCapactiy = 100
>
> agent1.channels.UserActionChannel.type = memory
> agent1.channels.UserActionChannel.capacity = 1000
> agent1.channels.UserActionChannel.transactionCapactiy = 100
>
> agent1.channels.VisitChannel.type = memory
> agent1.channels.VisitChannel.capacity = 1000
> agent1.channels.VisitChannel.transactionCapactiy = 100
>
> # Bind the source and sink to the channel
> agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel VisitChannel NullChannel