Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> why flume agent do not support --fileheader


+
周梦想 2013-02-22, 03:06
+
Juhani Connolly 2013-02-22, 04:08
Copy link to this message
-
Re: why flume agent do not support --fileheader
Yes,I mean all event headers send from a specific agent. some headers need
for all events, not only hostname. now I have to configure it on every
source time and time again.
for example:
if i configure 10 sources, every source need 5 header,  I have to configure
50 times.
other configure of the same type source have the same problem.

eg:
agent1.sources =gamelog src1 src2 ... src10

agent1.sources.gamelog.fileSuffix=.fin
agent1.sources.gamelog.fileHeader=true
agent1.sources.gamelog.fileHeaderKey=fullfilename
agent1.sources.gamelog.batchSize=100
agent1.sources.gamelog.bufferMaxLines=1000
agent1.sources.gamelog.bufferMaxLineLength=5000
agent1.sources.gamelog.interceptors = i1 i2 i3
#for %{host} org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.gamelog.interceptors.i1.type = host
agent1.sources.gamelog.interceptors.i2.type = timestamp
...
agent1.sources.gamelog.interceptors.i3.type = static
agent1.sources.gamelog.interceptors.i3.key = filename
agent1.sources.gamelog.interceptors.i3.value = gamelog

*repleat configure for src1,...src10 for 10 or even more times.*

it's very boring.
maybe we could group the sources,sinks and configure them one or several
times.

Best Regards,
Andy
2013/2/22 Juhani Connolly <[EMAIL PROTECTED]>

> If I understand your question, using a static interceptor is the expected
> way to do this.
>
> Not sure what you mean by agent header? Do you mean all event headers sent
> from a specific agent? I don't imagine we will be adding a command-line
> parameter to do this, it wouldn't be consistent and would be superfluous.
>
> If you want to reuse a configuration file, and add a header to inform what
> agent it came from, perhaps you could use the hostname interceptor?
>
>
> On 02/22/2013 12:06 PM, 周梦想 wrote:
>
>> I want add some key/value to the agent header, but it's not convenient to
>> do so. why not flume agent support --headerFile just like flume avro-client?
>>
>> my requirement is :
>> I want use spooling source to send files to another flume node to write
>> to hdfs.
>> and i want to name hdfs file to have the original file name prefix, not
>> the full path of the orgin file.
>>
>> now I have to add interceptors of conf to do that.
>>
>> so I have the question, why not flume agent support --headerFile just
>> like flume avro-client?
>>
>> Best Regards,
>> Andy
>>
>
>
+
Juhani Connolly 2013-02-22, 08:10
+
周梦想 2013-02-22, 08:36
+
Yogi Nerella 2013-02-22, 22:32
+
周梦想 2013-02-25, 03:23