Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> conf help needed


+
Cochran, David M 2012-08-23, 14:08
+
Bhaskar V. Karambelkar 2012-08-23, 16:17
Copy link to this message
-
Re: conf help needed
Hi David,

There's currently an open, unassigned ticket for what you're asking:
     FLUME-1295: RollingFileSink needs to be able to construct directory
path based on escape sequence

Cheers,
Will

On Thu, Aug 23, 2012 at 9:17 AM, Bhaskar V. Karambelkar <[EMAIL PROTECTED]
> wrote:

> Looking at the flume documentation, looks like dynamic paths are only
> supported for the HDFS sink and not the file_roll sink.
> You can use a Multiplexing channel selector to achieve the same effect for
> file_roll sink, but the possible hosts has to be a
> predetermined list and not dynamic.
>
>
> On Thu, Aug 23, 2012 at 10:08 AM, Cochran, David M (Contractor) <
> [EMAIL PROTECTED]> wrote:
>
>> **
>>
>> Let me see if I can explain what I'd like to accomplish, and yes I"m
>> certain I'm just not grasping something here and making a mess of it all,
>> and mayhaps ya'll can point me straight...
>>
>> Simple.. grab this log file xxx.log from host foo and store it in a on
>> the sink server as ~/flume/%host/xxx-%date.log
>>
>> Where the xxx.log and hostname are dynamic so multiple hosts/logs can be
>> templated.  I guess that makes sense... afterall that's what this whole
>> project is about...
>>
>> My troubles are coming from sorting the conf:
>>
>> here's where I've started... a combination of a bunch of conf's I've
>> borrowed bits from.
>>
>>
>> agent1.channels.ch1.type = memory
>>
>> agent1.sources =  tailsource-1
>> agent1.sources.channels = ch1
>> agent1.sources.tailsource-1.type = exec
>> agent1.sources.tailsource-1.command = tail -F
>> /root/Desktop/apache-flume-1.3.0-SNAPSHOT/test.log
>> agent1.sources.tailsource-1.channels = ch1
>> agent1.sources.tailsource1.bind = 0.0.0.0
>> agent1.sources.tailsource-1.port = 41414
>>
>> agent1.sources.tailsource-1.interceptors = hostint
>> agent1.sources.tailsource-1.interceptors.hostint.type >> org.apache.flume.interceptor.HostInterceptor$Builder
>> agent1.sources.tailsource-1.interceptors.hostint.preserveExisting = true
>> agent1.sources.tailsource-1.interceptors.hostint.useIP = false
>> agent1.sources.tailsource-1.interceptors.timestamp.type = timestamp
>>
>> agent1.sinks.log-sink1.channel = ch1
>> agent1.sinks.log-sink1.type = FILE_ROLL
>> agent1.sinks.log-sink1.sink.directory >> /root/Desktop/apache-flume-1.3.0-SNAPSHOT/logs/%{host}
>> agent1.sinks.log-sink1.sink.rollInterval = 86400
>> agent1.sinks.log-sink1.sink.filePrefix = %{filename}.%Y-%m-%d
>>
>>
>> Somehow I'm missing something as the %host never makes it forward, nor
>> does the log filename.... before adding the host/filename stuff in it
>> everything worked, just using filenames like 1345725771524-1 which aren't
>> very intuitive by themselves but lumped into an unnamed directory makes
>> them even less so.
>>
>>
>> This is the resulting logfile...
>>
>> 23 Aug 2012 08:29:49,146 ERROR
>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>> (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver
>> event. Exception follows.
>> org.apache.flume.EventDeliveryException: Failed to open file
>> /root/Desktop/apache-flume-1.3.0-SNAPSHOT/logs/%{host}/1345728588635-1
>> while delivering event
>>         at
>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:166)
>>         at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>         at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.FileNotFoundException:
>> /root/Desktop/apache-flume-1.3.0-SNAPSHOT/logs/%{host}/1345728588635-1 (No
>> such file or directory)
>>         at java.io.FileOutputStream.open(Native Method)
>>         at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
>>         at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
>>         at
>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:160)
>>         ... 3 more
>>
>>
>> Anyone care to knock some the the dummy-dust loose that's accumulated in
+
Cochran, David M 2012-09-05, 18:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB