Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> why flume agent do not support --fileheader


Copy link to this message
-
Re: why flume agent do not support --fileheader
Sounds like a job for an editor macro or a simple script.

Perhaps someone else can think of something else, but adding it to the
command line isn't feasible... If we started throwing random convenient
stuff there it would quickly become a mess

On 02/22/2013 01:30 PM, 周梦想 wrote:
> Yes,I mean all event headers send from a specific agent. some headers
> need for all events, not only hostname. now I have to configure it on
> every source time and time again.
> for example:
> if i configure 10 sources, every source need 5 header,  I have to
> configure 50 times.
> other configure of the same type source have the same problem.
>
> eg:
> agent1.sources =gamelog src1 src2 ... src10
>
> agent1.sources.gamelog.fileSuffix=.fin
> agent1.sources.gamelog.fileHeader=true
> agent1.sources.gamelog.fileHeaderKey=fullfilename
> agent1.sources.gamelog.batchSize=100
> agent1.sources.gamelog.bufferMaxLines=1000
> agent1.sources.gamelog.bufferMaxLineLength=5000
> agent1.sources.gamelog.interceptors = i1 i2 i3
> #for %{host} org.apache.flume.interceptor.HostInterceptor$Builder
> agent1.sources.gamelog.interceptors.i1.type = host
> agent1.sources.gamelog.interceptors.i2.type = timestamp
> ...
> agent1.sources.gamelog.interceptors.i3.type = static
> agent1.sources.gamelog.interceptors.i3.key = filename
> agent1.sources.gamelog.interceptors.i3.value = gamelog
>
> *repleat configure for src1,...src10 for 10 or even more times.*
>
> it's very boring.
> maybe we could group the sources,sinks and configure them one or
> several times.
>
> Best Regards,
> Andy
>
>
> 2013/2/22 Juhani Connolly <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>>
>
>     If I understand your question, using a static interceptor is the
>     expected way to do this.
>
>     Not sure what you mean by agent header? Do you mean all event
>     headers sent from a specific agent? I don't imagine we will be
>     adding a command-line parameter to do this, it wouldn't be
>     consistent and would be superfluous.
>
>     If you want to reuse a configuration file, and add a header to
>     inform what agent it came from, perhaps you could use the hostname
>     interceptor?
>
>
>     On 02/22/2013 12:06 PM, 周梦想 wrote:
>
>         I want add some key/value to the agent header, but it's not
>         convenient to do so. why not flume agent support --headerFile
>         just like flume avro-client?
>
>         my requirement is :
>         I want use spooling source to send files to another flume node
>         to write to hdfs.
>         and i want to name hdfs file to have the original file name
>         prefix, not the full path of the orgin file.
>
>         now I have to add interceptors of conf to do that.
>
>         so I have the question, why not flume agent support
>         --headerFile just like flume avro-client?
>
>         Best Regards,
>         Andy
>
>
>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB