Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> about flume-ng agent


Copy link to this message
-
RE: about flume-ng agent
I have tried to go the syslogUDP route to get log files from a Windows server to a flume agent, and did not find it an adequate solution.

- We are seeing corruped events when sending IIS logs (known issue: https://issues.apache.org/jira/browse/FLUME-1365)
- Our data is too large to fit in a 1500 byte ethernet frame so events are fragmented and the syslogUDP source ignores the continuation packets, resulting in truncated events.

I have just been able to build the flume-ng agent for Windows and am testing the avro-client functionality on Windows. I think this will be the best bet for us in the short term, using LogParser to incrementally create files via scheduled task for the avro client to send along.

Long term we want to develop a .net Avro client library for our apps to use directly. I suppose a log4net avro appender would be nice too.
-----Original Message-----
From: Brock Noland [mailto:[EMAIL PROTECTED]]
Sent: Thursday, October 25, 2012 9:22 AM
To: [EMAIL PROTECTED]
Subject: Re: about flume-ng agent

If you cannot use RPCclient (project is not in java), then writing the events to syslog and then sending those events to a "collector" agent running syslog source is probably the best option. A worse option would be to use exec source with tail -F. This is "worse" because it can easily lose large amounts of data.

Brock

On Thu, Oct 25, 2012 at 11:00 AM, lancexxx <[EMAIL PROTECTED]> wrote:
> oh, seemingly ,I see. sorry , I am new to flume.
> now I collect log from web server and want to use syslogudp source,
> which tool or  RPCclient  I should use to sent the data to the source
> of flume-ng agent on web server host ? maybe can you recommend to me a
> better source type like AVRO source, syslog source etc. because I do
> not realized the difference or advantage between them and I find no
> more information via the official guide。
> thanks very much!
> --
> lancexxx
>
> On 2012年10月25日Thursday at 下午10:37, Brock Noland wrote:
>
> Either the webserver must run a flume agent, the webserver must use
> the RPCClient (just a java object, not an agent) or the webserver can
> use the log4j appender.
>
> Brock
>
> On Wed, Oct 24, 2012 at 10:51 PM, lancexxx <[EMAIL PROTECTED]> wrote:
>
>
> hi
> I do not understand that every host of webserser must run a flume-ng
> agent if I collect weblog?
> if no ,well then the client(web server host) how to sent the log to
> the flume-ng agent host in the internet?
> --
> thanks!
> lancexxx
>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/