Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> flume to collect log files from various sources

Copy link to this message
Re: flume to collect log files from various sources
Hello Ramya,

Flume agent have to be installed on every host?
    I want to avoid this by integrating log4jappender into my app, and to
remove one additional overhead for administrators.
    Is this a problem?
Collecting data using "taildir" command?
   1.   If I have one flume agent sending log files from one host to
another agent who is listening.
    2.   How do I configure the receiving agent such that, it can separate
the log files again and write as it recieved.
Any sample configuration files for either 'taildir'  or 'multitail'

Issue with log4jappender:

I am evaluating log4jappender, and using this I am not able to send events
to the flume agent.

When I turn on debugging all the log messages I see are as follows.

14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
14:27:29,604  WARN NettyAvroRpcClient:426 - Invalid value for batchSize: 0;
Using default value.
14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
{tcpNoDelay=true, connectTimeoutMillis=20000}
14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, / => /] BOUND: /
log4j:ERROR Flume append() failed.
14:27:29,940  INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST 2013
14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from

On Wed, Jan 23, 2013 at 8:57 PM, Sri Ramya <[EMAIL PROTECTED]> wrote:

> there is an error displayed on console Flume append() failed. Try to
> resolve .
> explain me what is your exact problem.
> Note:
> 1, Flume agent have to install in every host from where u want to collect
> data.
> 2, If u want to collect data from a directory then you have to to use
> 'taildir' command.
> 3, If u want to collect data from different files u have to use
> 'multitail' command.
> I think this info will be use full for u.
> thank you
> On Thu, Jan 24, 2013 at 4:06 AM, yogi nerella <[EMAIL PROTECTED]>wrote:
>> Hi,
>> I am a new bee for flume, would like to get some feedback on if what I am
>> doing works correctly.
>> My application runs on multiple hosts, I want to collect all log files to
>> a central location.
>> 1.  In My application I will ship all relevant log4jappender jar files,
>> and configure the agent's host, port information.
>> 2. I will run a simple agent, and configure a source (avro), channel
>> (memory), and sink (file_roll).
>> File_roll will write the events recieved via AVRO source and write them
>> to the corresponding file_roll sink.
>> 3. Do I need one agent for each host, or is there a way I can configure
>> to write to multiple files, dynamically based on the real source the
>> message is getting recieved.
>> 4. If I modify the configuration file, will the agent re-read the