Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> about flume-ng agent


Copy link to this message
-
RE: about flume-ng agent
Answers inline:

>On Thu, Oct 25, 2012 at 11:54 AM, Paul Chavez <[EMAIL PROTECTED]> wrote:
>> I have tried to go the syslogUDP route to get log files from a Windows server to a flume agent, and did not find it an adequate solution.
>>
>> - We are seeing corruped events when sending IIS logs (known issue:
>> https://issues.apache.org/jira/browse/FLUME-1365)
>> - Our data is too large to fit in a 1500 byte ethernet frame so events are fragmented and the syslogUDP source ignores the continuation packets, resulting in truncated events.
>
>I am not a Syslog expert, is there a way to detect the continuation packets? If so I think we should open a JIRA on this issue.

I am no expert either but the various syslog related RFC and RFC-type documentation I can find recommends that messeages be kept small in order to avoid fragmentation.

>From an IP perspective, there are Ipv4 header flags that indicate if more fragments follow along with a fragment identifier to help reassemble out-of-order packets so it certainly is possible even if it goes against RFC recommendations.

Testing with the syslogTCP source did not show any issues with fragmentation, but the tool we are using to send syslog messages over TCP (LogParser) does not separate messages with a carriage return so messages weren't parsed correctly by the flume source.

>> I have just been able to build the flume-ng agent for Windows and am testing the avro-client functionality on Windows. I think this will be the best bet for us in the short term, using LogParser to incrementally create files via scheduled task for the avro client to send along.
>>
>> Long term we want to develop a .net Avro client library for our apps to use directly. I suppose a log4net avro appender would be nice too.
>
>I think a .NET version of the RPCClient and log4net appender would be awesome.  If you do write them, would you considering pushing them back to the flume community?

I would hope so but I am not in a position to make any guarantees on behalf of my employer.