Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> issue with hdfs sink config


Copy link to this message
-
Re: issue with hdfs sink config
Raymond,

I don't think you need the port 50070 on your hdfs path.
Try setting this to the fqdn of your hdfs namenode, leave off the port
50070 (this is just for the webui)

-Jeff

On Thu, Sep 13, 2012 at 9:04 AM, Raymond Ng <[EMAIL PROTECTED]> wrote:

> Hi
>
> I'm baffled as to what the problem is with my hdfs config below
>
> agent.sinks.probeHdfsSink2.type = hdfs
> agent.sinks.probeHdfsSink2.hdfs.path > hdfs://master:50070/user/test/data/flume/%{client}
> agent.sinks.probeHdfsSink2.hdfs.filePrefix = %{site}_sessionData_%Y%m%d
> agent.sinks.probeHdfsSink2.channel = probeFileChannel2
> agent.sinks.probeHdfsSink2.hdfs.rollInterval = 120
> agent.sinks.probeHdfsSink2.hdfs.rollSize = 209715200
> agent.sinks.probeHdfsSink2.hdfs.rollCount = 0
> agent.sinks.probeHdfsSink2.hdfs.batchSize = 100000
> agent.sinks.probeHdfsSink2.hdfs.writeFormat = Text
> agent.sinks.probeHdfsSink2.hdfs.fileType = DataStream
> the flume log below does not provide enough info to help identify the
> problem, FlumeConfiguration.AgentConfiguration holds a list of
> <FlumeConfigurationError> but none of them get displayed in the log.
>
> If I replace hdfs sink with a file sink then everything works, so the
> problem is definitely in my hdfs configs but I can't see where?  any advice
> is appreciated
>
> 2012-09-13 16:49:43,145  INFO [main] LifecycleSupervisor.java - Starting
> lifecycle supervisor 1
> 2012-09-13 16:49:43,151  INFO [main] FlumeNode.java - Flume node starting
> - agent
> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-0]
> DefaultLogicalNodeManager.java - Node manager starting
> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-2]
> AbstractFileConfigurationProvider.java - Configuration provider starting
> 2012-09-13 16:49:43,157  INFO [lifecycleSupervisor-1-0]
> LifecycleSupervisor.java - Starting lifecycle supervisor 11
> 2012-09-13 16:49:43,158 DEBUG [lifecycleSupervisor-1-0]
> DefaultLogicalNodeManager.java - Node manager started
> 2012-09-13 16:49:43,160 DEBUG [lifecycleSupervisor-1-2]
> AbstractFileConfigurationProvider.java - Configuration provider started
> 2012-09-13 16:49:43,160 DEBUG [conf-file-poller-0]
> AbstractFileConfigurationProvider.java - Checking
> file:/usr/local/lib/flume-ng/conf/acme-agent.conf for changes
> 2012-09-13 16:49:43,160  INFO [conf-file-poller-0]
> AbstractFileConfigurationProvider.java - Reloading configuration
> file:/usr/local/lib/flume-ng/conf/acme-agent.conf
> 2012-09-13 16:49:43,170  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Added sinks: probeHdfsSink2 Agent: agent
> 2012-09-13 16:49:43,171  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,171 DEBUG [conf-file-poller-0] FlumeConfiguration.java
> - Created context for probeHdfsSink2: hdfs.rollSize
> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeFileSink
> 2012-09-13 16:49:43,172 DEBUG [conf-file-poller-0] FlumeConfiguration.java
> - Created context for probeFileSink: sink.rollInterval
> 2012-09-13 16:49:43,176  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,176  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeFileSink
> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeFileSink
> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeFileSink
> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0] FlumeConfiguration.java