Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - issue with hdfs sink config
Raymond Ng
2012-09-13, 16:04
Jeff Lord
2012-09-13, 16:28
Raymond Ng
2012-09-14, 07:35
Raymond Ng
2012-09-15, 06:28
+
Raymond Ng 2012-09-13, 16:04
+
Jeff Lord 2012-09-13, 16:28
+
Raymond Ng 2012-09-14, 07:35
Copy link to this message
-
Re: issue with hdfs sink config
Jeff Lord 2012-09-14, 21:16
Let's backup a step.
Do you have the hadoop libs installed?
In this version you need to have the hadoop libs installed in order for
flume to write to hdfs sink.
On Fri, Sep 14, 2012 at 12:35 AM, Raymond Ng <[EMAIL PROTECTED]> wrote:

> Hi Jeff
>
> removing the port number didn't work
>
> what I should have mentioned was that I'm using flume 1.2, the same config
> I mentioned below works for flume 1.1, so I'm not sure what has changed in
> 1.2 for the configs and I need to use 1.2 because of the File channel
>  I did see HDFSSinkConfiguration mentioned in SinkConfiguration.java but
> I can't find this class in the flume 1.2 jars, and ClassNotFoundException
> is being wrapped up in ConfigurationException which doesn't get reported in
> the log
>
>
> Ray
>
>
> On Thu, Sep 13, 2012 at 5:28 PM, Jeff Lord <[EMAIL PROTECTED]> wrote:
>
>> Raymond,
>>
>> I don't think you need the port 50070 on your hdfs path.
>> Try setting this to the fqdn of your hdfs namenode, leave off the port
>> 50070 (this is just for the webui)
>>
>> -Jeff
>>
>> On Thu, Sep 13, 2012 at 9:04 AM, Raymond Ng <[EMAIL PROTECTED]> wrote:
>>
>>> Hi
>>>
>>> I'm baffled as to what the problem is with my hdfs config below
>>>
>>> agent.sinks.probeHdfsSink2.type = hdfs
>>> agent.sinks.probeHdfsSink2.hdfs.path >>> hdfs://master:50070/user/test/data/flume/%{client}
>>> agent.sinks.probeHdfsSink2.hdfs.filePrefix = %{site}_sessionData_%Y%m%d
>>> agent.sinks.probeHdfsSink2.channel = probeFileChannel2
>>> agent.sinks.probeHdfsSink2.hdfs.rollInterval = 120
>>> agent.sinks.probeHdfsSink2.hdfs.rollSize = 209715200
>>> agent.sinks.probeHdfsSink2.hdfs.rollCount = 0
>>> agent.sinks.probeHdfsSink2.hdfs.batchSize = 100000
>>> agent.sinks.probeHdfsSink2.hdfs.writeFormat = Text
>>> agent.sinks.probeHdfsSink2.hdfs.fileType = DataStream
>>> the flume log below does not provide enough info to help identify the
>>> problem, FlumeConfiguration.AgentConfiguration holds a list of
>>> <FlumeConfigurationError> but none of them get displayed in the log.
>>>
>>> If I replace hdfs sink with a file sink then everything works, so the
>>> problem is definitely in my hdfs configs but I can't see where?  any advice
>>> is appreciated
>>>
>>> 2012-09-13 16:49:43,145  INFO [main] LifecycleSupervisor.java - Starting
>>> lifecycle supervisor 1
>>> 2012-09-13 16:49:43,151  INFO [main] FlumeNode.java - Flume node
>>> starting - agent
>>> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-0]
>>> DefaultLogicalNodeManager.java - Node manager starting
>>> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-2]
>>> AbstractFileConfigurationProvider.java - Configuration provider starting
>>> 2012-09-13 16:49:43,157  INFO [lifecycleSupervisor-1-0]
>>> LifecycleSupervisor.java - Starting lifecycle supervisor 11
>>> 2012-09-13 16:49:43,158 DEBUG [lifecycleSupervisor-1-0]
>>> DefaultLogicalNodeManager.java - Node manager started
>>> 2012-09-13 16:49:43,160 DEBUG [lifecycleSupervisor-1-2]
>>> AbstractFileConfigurationProvider.java - Configuration provider started
>>> 2012-09-13 16:49:43,160 DEBUG [conf-file-poller-0]
>>> AbstractFileConfigurationProvider.java - Checking
>>> file:/usr/local/lib/flume-ng/conf/acme-agent.conf for changes
>>> 2012-09-13 16:49:43,160  INFO [conf-file-poller-0]
>>> AbstractFileConfigurationProvider.java - Reloading configuration
>>> file:/usr/local/lib/flume-ng/conf/acme-agent.conf
>>> 2012-09-13 16:49:43,170  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Added sinks: probeHdfsSink2 Agent: agent
>>> 2012-09-13 16:49:43,171  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,171 DEBUG [conf-file-poller-0]
>>> FlumeConfiguration.java - Created context for probeHdfsSink2: hdfs.rollSize
>>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
+
Raymond Ng 2012-09-15, 06:28