Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - flume-ng syslogtcp issue


Copy link to this message
-
Re: flume-ng syslogtcp issue
Juhani Connolly 2012-07-12, 06:15
file_roll is detailed in the Flume user guide. It's generated in
target/docs when you do a full build. I've put it up at
http://people.apache.org/~juhanic/flume-docs/FlumeUserGuide.html#file-roll-sink
<http://people.apache.org/%7Ejuhanic/flume-docs/FlumeUserGuide.html#file-roll-sink>
for convenience, but the version may not match yours.

If the docs are insufficient let us know and we'll try to address it.

On 07/12/2012 02:38 PM, Young Kim wrote:
> After some debugging, I found out that the syslogTcp is working as
> intended. I didn't realize that debug no longer shows the resulting
> syslog messages on success. On that note, this may be a bit off-topic,
> but is there documentation showing how file_roll should be used to
> write to local disk?
>
> Thanks,
> Young
>
> On Wednesday, July 11, 2012 at 11:12 AM, Young Kim wrote:
>
>> Hey Alex,
>>
>> The IP is correct, but I've altered it to say localhost. As for the
>> syslog tests, I've ran it verbatim, but it doesn't register. However,
>> when I alter /tmp/foo to include two new lines, flume registers it as
>> a partial event.
>>
>> It still seems odd to me that flume-ng isn't detecting the syslogtcp
>> whereas flume-og is doing it perfectly. Is it possible that there is
>> a bug in the way flume-ng is implementing it?
>>
>> Thanks,
>> Young
>>
>> On Tuesday, July 10, 2012 at 11:22 PM, alo alt wrote:
>>
>>> HI,
>>>
>>> try these for syslog tests:
>>> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
>>> nc -v FLUME SYSLOG AGENT 5140 < /tmp/foo
>>>
>>>> syslog-agent.sources.syslog.host = 172.17.1.92
>>>
>>> Means, will only listen only on this interface, is the IP correct?
>>>
>>> cheers,
>>> Alex
>>>
>>> On Jul 10, 2012, at 10:59 PM, Young Kim wrote:
>>>
>>>> Hey guys,
>>>>
>>>> I've been trying to use flume-ng (build in trunk) to read syslogs
>>>> through tcp then write to a flat file, as seen in this configuration:
>>>>
>>>> # Source, sink, and channel name below
>>>> syslog-agent.channels = rmc
>>>> syslog-agent.sources = syslog
>>>> syslog-agent.sinks = flat
>>>>
>>>> # Configure channels
>>>> syslog-agent.channels.rmc.type = memory
>>>>
>>>> # Configure sources
>>>> syslog-agent.sources.syslog.type = syslogTcp
>>>> syslog-agent.sources.syslog.port = 5140
>>>> syslog-agent.sources.syslog.host = 172.17.1.92
>>>> syslog-agent.sources.syslog.channels = rmc
>>>>
>>>> # Configure sinks
>>>> syslog-agent.sinks.flat.type = FILE_ROLL
>>>> syslog-agent.sinks.flat.directory = /Users/ykim/Desktop/test
>>>> syslog-agent.sinks.flat.rollInterval = 0
>>>> syslog-agent.sinks.flat.channels = rmc
>>>>
>>>>
>>>> After a bit of wrangling with rsyslog configurations, it appeared
>>>> as if rsyslog wasn't sending any packets. However, I decided to try
>>>> out flume-og with a simple dump syslogTcp, and it correctly reads
>>>> the syslog. At this point, I'm a bit lost and not sure what else to
>>>> look at. Hopefully, someone on here can shed some light on what
>>>> could be wrong.
>>>>
>>>> Thanks,
>>>> Young Kim
>>>
>>>
>>> --
>>> Alexander Alten-Lorenz
>>> http://mapredit.blogspot.com
>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>