Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> about flume-ng agent


Copy link to this message
-
Re: about flume-ng agent
Hi,

Questions inline:

On Thu, Oct 25, 2012 at 11:54 AM, Paul Chavez
<[EMAIL PROTECTED]> wrote:
> I have tried to go the syslogUDP route to get log files from a Windows server to a flume agent, and did not find it an adequate solution.
>
> - We are seeing corruped events when sending IIS logs (known issue: https://issues.apache.org/jira/browse/FLUME-1365)
> - Our data is too large to fit in a 1500 byte ethernet frame so events are fragmented and the syslogUDP source ignores the continuation packets, resulting in truncated events.

I am not a Syslog expert, is there a way to detect the continuation
packets? If so I think we should open a JIRA on this issue.
> I have just been able to build the flume-ng agent for Windows and am testing the avro-client functionality on Windows. I think this will be the best bet for us in the short term, using LogParser to incrementally create files via scheduled task for the avro client to send along.
>
> Long term we want to develop a .net Avro client library for our apps to use directly. I suppose a log4net avro appender would be nice too.

I think a .NET version of the RPCClient and log4net appender would be
awesome.  If you do write them, would you considering pushing them
back to the flume community?

Brock
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB