Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> conf help needed


+
Cochran, David M 2012-08-23, 14:08
+
Bhaskar V. Karambelkar 2012-08-23, 16:17
Copy link to this message
-
Re: conf help needed
Hi David,

There's currently an open, unassigned ticket for what you're asking:
     FLUME-1295: RollingFileSink needs to be able to construct directory
path based on escape sequence

Cheers,
Will

On Thu, Aug 23, 2012 at 9:17 AM, Bhaskar V. Karambelkar <[EMAIL PROTECTED]
> wrote:

> Looking at the flume documentation, looks like dynamic paths are only
> supported for the HDFS sink and not the file_roll sink.
> You can use a Multiplexing channel selector to achieve the same effect for
> file_roll sink, but the possible hosts has to be a
> predetermined list and not dynamic.
>
>
> On Thu, Aug 23, 2012 at 10:08 AM, Cochran, David M (Contractor) <
> [EMAIL PROTECTED]> wrote:
>
>> **
>>
>> Let me see if I can explain what I'd like to accomplish, and yes I"m
>> certain I'm just not grasping something here and making a mess of it all,
>> and mayhaps ya'll can point me straight...
>>
>> Simple.. grab this log file xxx.log from host foo and store it in a on
>> the sink server as ~/flume/%host/xxx-%date.log
>>
>> Where the xxx.log and hostname are dynamic so multiple hosts/logs can be
>> templated.  I guess that makes sense... afterall that's what this whole
>> project is about...
>>
>> My troubles are coming from sorting the conf:
>>
>> here's where I've started... a combination of a bunch of conf's I've
>> borrowed bits from.
>>
>>
>> agent1.channels.ch1.type = memory
>>
>> agent1.sources =  tailsource-1
>> agent1.sources.channels = ch1
>> agent1.sources.tailsource-1.type = exec
>> agent1.sources.tailsource-1.command = tail -F
>> /root/Desktop/apache-flume-1.3.0-SNAPSHOT/test.log
>> agent1.sources.tailsource-1.channels = ch1
>> agent1.sources.tailsource1.bind = 0.0.0.0
>> agent1.sources.tailsource-1.port = 41414
>>
>> agent1.sources.tailsource-1.interceptors = hostint
>> agent1.sources.tailsource-1.interceptors.hostint.type >> org.apache.flume.interceptor.HostInterceptor$Builder
>> agent1.sources.tailsource-1.interceptors.hostint.preserveExisting = true
>> agent1.sources.tailsource-1.interceptors.hostint.useIP = false
>> agent1.sources.tailsource-1.interceptors.timestamp.type = timestamp
>>
>> agent1.sinks.log-sink1.channel = ch1
>> agent1.sinks.log-sink1.type = FILE_ROLL
>> agent1.sinks.log-sink1.sink.directory >> /root/Desktop/apache-flume-1.3.0-SNAPSHOT/logs/%{host}
>> agent1.sinks.log-sink1.sink.rollInterval = 86400
>> agent1.sinks.log-sink1.sink.filePrefix = %{filename}.%Y-%m-%d
>>
>>
>> Somehow I'm missing something as the %host never makes it forward, nor
>> does the log filename.... before adding the host/filename stuff in it
>> everything worked, just using filenames like 1345725771524-1 which aren't
>> very intuitive by themselves but lumped into an unnamed directory makes
>> them even less so.
>>
>>
>> This is the resulting logfile...
>>
>> 23 Aug 2012 08:29:49,146 ERROR
>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>> (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver
>> event. Exception follows.
>> org.apache.flume.EventDeliveryException: Failed to open file
>> /root/Desktop/apache-flume-1.3.0-SNAPSHOT/logs/%{host}/1345728588635-1
>> while delivering event
>>         at
>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:166)
>>         at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>         at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.FileNotFoundException:
>> /root/Desktop/apache-flume-1.3.0-SNAPSHOT/logs/%{host}/1345728588635-1 (No
>> such file or directory)
>>         at java.io.FileOutputStream.open(Native Method)
>>         at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
>>         at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
>>         at
>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:160)
>>         ... 3 more
>>
>>
>> Anyone care to knock some the the dummy-dust loose that's accumulated in
+
Cochran, David M 2012-09-05, 18:29