Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Error: Flume to HDFS Sink


+
Rajesh Jain 2013-07-17, 20:47
+
Paul Chavez 2013-07-17, 21:03
Copy link to this message
-
Re: Error: Flume to HDFS Sink
hi Paul

This did the trick.
agent1.channels.NullChannel.type = memory

Thanks
On Wed, Jul 17, 2013 at 5:03 PM, Paul Chavez <
[EMAIL PROTECTED]> wrote:

> **
> You've specified a channel named NullChannel in the agent1.channels
> properties, but you don't define it anywhere in the configuration. You've
> done the same with the NullSink.
>
> You'll need to add something like this to the config:
>
> agent1.channels.NullChannel.type = memory
>
> agent1.sinks.NullSink.type = null
>
> Hope that helps,
> Paul
>
>
>  ------------------------------
> *From:* Rajesh Jain [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, July 17, 2013 1:47 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Error: Flume to HDFS Sink
>
>  I am getting a strange error when I am configuring Flume to write to a
> HDFS Sink.
>
>  17 Jul 2013 20:36:46,469 ERROR [conf-file-poller-0]
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
>  - Failed to load configuration data. Exception follows.
> org.apache.flume.FlumeException: Selector channel not found: *NullChannel*
>         at
> org.apache.flume.channel.MultiplexingChannelSelector.getChannelListFromNames(MultiplexingChannelSelector.java:166)
>
>
> Do I need to change my configure file, here is my flume-conf.properties
>
> Second question, the hdfs.path - does it have to be hdfs namenode syntax
>
> I am new to both flume and hadoop/hdfs.
>
>
>
> # Name the components on this agent
> agent1.sources = HTTPSource
> agent1.sinks = PurePathSink UserActionSink VisitSink NullSink
> agent1.channels = PurePathChannel UserActionChannel VisitChannel *NullChannel*
>
> # Describe/configure HTTPSource
> agent1.sources.HTTPSource.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.HTTPSource.port = 4321
> agent1.sources.HTTPSource.handler = com.dynatrace.diagnostics.btexport.flume.BtExportHandler
>
> # Describe sinks
> agent1.sinks.PurePathSink.type = hdfs
> agent1.sinks.PurePathSink.hdfs.path = hdfs://localhost:9000/user/bts/pp
> agent1.sinks.PurePathSink.hdfs.fileType = DataStream
> agent1.sinks.PurePathSink.hdfs.filePrefix = export
> agent1.sinks.PurePathSink.hdfs.fileSuffix = .txt
> agent1.sinks.PurePathSink.hdfs.rollInterval = 120
> agent1.sinks.PurePathSink.hdfs.rollSize = 131072
> agent1.sinks.PurePathSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializerBuilder
>
> agent1.sinks.UserActionSink.type = hdfs
> agent1.sinks.UserActionSink.hdfs.path = hdfs://localhost:9000/user/bts/pa
> agent1.sinks.UserActionSink.hdfs.fileType = DataStream
> agent1.sinks.UserActionSink.hdfs.filePrefix = export
> agent1.sinks.UserActionSink.hdfs.fileSuffix = .txt
> agent1.sinks.UserActionSink.hdfs.rollInterval = 120
> agent1.sinks.UserActionSink.hdfs.rollSize = 131072
> agent1.sinks.UserActionSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPageActionSerializerBuilder
>
> agent1.sinks.VisitSink.type = hdfs
> agent1.sinks.VisitSink.hdfs.path = hdfs://localhost:9000/user/bts/visit
> agent1.sinks.VisitSink.hdfs.fileType = DataStream
> agent1.sinks.VisitSink.hdfs.filePrefix = export
> agent1.sinks.VisitSink.hdfs.fileSuffix = .txt
> agent1.sinks.VisitSink.hdfs.rollInterval = 120
> agent1.sinks.VisitSink.hdfs.rollSize = 131072
> agent1.sinks.VisitSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtVisitSerializerBuilder
>
> agent1.sinks.NullSink.type = null
>
> # Use a channel which buffers events in memory
> agent1.channels.PurePathChannel.type = memory
> agent1.channels.PurePathChannel.capacity = 1000
> agent1.channels.PurePathChannel.transactionCapactiy = 100
>
> agent1.channels.UserActionChannel.type = memory
> agent1.channels.UserActionChannel.capacity = 1000
> agent1.channels.UserActionChannel.transactionCapactiy = 100
>
> agent1.channels.VisitChannel.type = memory
> agent1.channels.VisitChannel.capacity = 1000
> agent1.channels.VisitChannel.transactionCapactiy = 100
>
> # Bind the source and sink to the channel
> agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel VisitChannel NullChannel
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB