Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - why flume agent do not support --fileheader


Copy link to this message
-
Re: why flume agent do not support --fileheader
Yogi Nerella 2013-02-22, 22:32
Hi,

I want to ship various types of files from one agent to another agent, and
a destination agent should rewrite them to the same filename as the source
filename.

Like a simple FTP source to FTP destination, but I need it via flume
because I want to do some intercepting and processing?   Is there any
simple configuration for it?   Or Flume cannot support such functionality?

Thanks,
Yogi
On Fri, Feb 22, 2013 at 12:36 AM, 周梦想 <[EMAIL PROTECTED]> wrote:

> ok,I see.
>
>
> 2013/2/22 Juhani Connolly <[EMAIL PROTECTED]>
>
>>  Sounds like a job for an editor macro or a simple script.
>>
>> Perhaps someone else can think of something else, but adding it to the
>> command line isn't feasible... If we started throwing random convenient
>> stuff there it would quickly become a mess
>>
>>
>> On 02/22/2013 01:30 PM, 周梦想 wrote:
>>
>> Yes,I mean all event headers send from a specific agent. some headers
>> need for all events, not only hostname. now I have to configure it on every
>> source time and time again.
>> for example:
>> if i configure 10 sources, every source need 5 header,  I have to
>> configure 50 times.
>> other configure of the same type source have the same problem.
>>
>>  eg:
>> agent1.sources =gamelog src1 src2 ... src10
>>
>>  agent1.sources.gamelog.fileSuffix=.fin
>> agent1.sources.gamelog.fileHeader=true
>> agent1.sources.gamelog.fileHeaderKey=fullfilename
>> agent1.sources.gamelog.batchSize=100
>> agent1.sources.gamelog.bufferMaxLines=1000
>> agent1.sources.gamelog.bufferMaxLineLength=5000
>>  agent1.sources.gamelog.interceptors = i1 i2 i3
>> #for %{host} org.apache.flume.interceptor.HostInterceptor$Builder
>> agent1.sources.gamelog.interceptors.i1.type = host
>>  agent1.sources.gamelog.interceptors.i2.type = timestamp
>> ...
>>  agent1.sources.gamelog.interceptors.i3.type = static
>> agent1.sources.gamelog.interceptors.i3.key = filename
>> agent1.sources.gamelog.interceptors.i3.value = gamelog
>>
>>  *repleat configure for src1,...src10 for 10 or even more times.*
>>
>>  it's very boring.
>> maybe we could group the sources,sinks and configure them one or several
>> times.
>>
>>  Best Regards,
>> Andy
>>
>>
>> 2013/2/22 Juhani Connolly <[EMAIL PROTECTED]>
>>
>>> If I understand your question, using a static interceptor is the
>>> expected way to do this.
>>>
>>> Not sure what you mean by agent header? Do you mean all event headers
>>> sent from a specific agent? I don't imagine we will be adding a
>>> command-line parameter to do this, it wouldn't be consistent and would be
>>> superfluous.
>>>
>>> If you want to reuse a configuration file, and add a header to inform
>>> what agent it came from, perhaps you could use the hostname interceptor?
>>>
>>>
>>> On 02/22/2013 12:06 PM, 周梦想 wrote:
>>>
>>>> I want add some key/value to the agent header, but it's not convenient
>>>> to do so. why not flume agent support --headerFile just like flume
>>>> avro-client?
>>>>
>>>> my requirement is :
>>>> I want use spooling source to send files to another flume node to write
>>>> to hdfs.
>>>> and i want to name hdfs file to have the original file name prefix, not
>>>> the full path of the orgin file.
>>>>
>>>> now I have to add interceptors of conf to do that.
>>>>
>>>> so I have the question, why not flume agent support --headerFile just
>>>> like flume avro-client?
>>>>
>>>> Best Regards,
>>>> Andy
>>>>
>>>
>>>
>>
>>
>