Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> flume-ng syslogtcp issue


+
Young Kim 2012-07-10, 20:59
+
alo alt 2012-07-11, 06:22
+
Young Kim 2012-07-11, 18:12
+
Young Kim 2012-07-12, 05:38
Copy link to this message
-
Re: flume-ng syslogtcp issue
Hi Young,
we've significantly improved user guide recently - you can get updated version either in repository (trunk) or in release 1.2.0 that is currently under vote (not released yet). Example of file_roll usage is present there.

Jarcec

On Jul 12, 2012, at 7:38 AM, Young Kim wrote:

> After some debugging, I found out that the syslogTcp is working as intended. I didn't realize that debug no longer shows the resulting syslog messages on success. On that note, this may be a bit off-topic, but is there documentation showing how file_roll should be used to write to local disk?
>
> Thanks,
> Young
> On Wednesday, July 11, 2012 at 11:12 AM, Young Kim wrote:
>
>> Hey Alex,
>>
>> The IP is correct, but I've altered it to say localhost. As for the syslog tests, I've ran it verbatim, but it doesn't register. However, when I alter /tmp/foo to include two new lines, flume registers it as a partial event.
>>
>> It still seems odd to me that flume-ng isn't detecting the syslogtcp whereas flume-og is doing it perfectly. Is it possible that there is a bug in the way flume-ng is implementing it?
>>
>> Thanks,
>> Young
>>
>> On Tuesday, July 10, 2012 at 11:22 PM, alo alt wrote:
>>
>>> HI,
>>>
>>> try these for syslog tests:
>>> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
>>> nc -v FLUME SYSLOG AGENT 5140 < /tmp/foo
>>>
>>>> syslog-agent.sources.syslog.host = 172.17.1.92
>>>
>>> Means, will only listen only on this interface, is the IP correct?
>>>
>>> cheers,
>>> Alex
>>>
>>> On Jul 10, 2012, at 10:59 PM, Young Kim wrote:
>>>
>>>> Hey guys,
>>>>
>>>> I've been trying to use flume-ng (build in trunk) to read syslogs through tcp then write to a flat file, as seen in this configuration:
>>>>
>>>> # Source, sink, and channel name below
>>>> syslog-agent.channels = rmc
>>>> syslog-agent.sources = syslog
>>>> syslog-agent.sinks = flat
>>>>
>>>> # Configure channels
>>>> syslog-agent.channels.rmc.type = memory
>>>>
>>>> # Configure sources
>>>> syslog-agent.sources.syslog.type = syslogTcp
>>>> syslog-agent.sources.syslog.port = 5140
>>>> syslog-agent.sources.syslog.host = 172.17.1.92
>>>> syslog-agent.sources.syslog.channels = rmc
>>>>
>>>> # Configure sinks
>>>> syslog-agent.sinks.flat.type = FILE_ROLL
>>>> syslog-agent.sinks.flat.directory = /Users/ykim/Desktop/test
>>>> syslog-agent.sinks.flat.rollInterval = 0
>>>> syslog-agent.sinks.flat.channels = rmc
>>>>
>>>>
>>>> After a bit of wrangling with rsyslog configurations, it appeared as if rsyslog wasn't sending any packets. However, I decided to try out flume-og with a simple dump syslogTcp, and it correctly reads the syslog. At this point, I'm a bit lost and not sure what else to look at. Hopefully, someone on here can shed some light on what could be wrong.
>>>>
>>>> Thanks,
>>>> Young Kim
>>>
>>>
>>> --
>>> Alexander Alten-Lorenz
>>> http://mapredit.blogspot.com
>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>

+
Juhani Connolly 2012-07-12, 06:15
+
Stern, Mark 2012-07-12, 06:21
+
Hari Shreedharan 2012-07-12, 06:27
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB