Hey guys,

I've been trying to use flume-ng (build in trunk) to read syslogs through tcp then write to a flat file, as seen in this configuration:

# Source, sink, and channel name below
syslog-agent.channels = rmc
syslog-agent.sources = syslog
syslog-agent.sinks = flat

# Configure channels
syslog-agent.channels.rmc.type = memory

# Configure sources
syslog-agent.sources.syslog.type = syslogTcp
syslog-agent.sources.syslog.port = 5140
syslog-agent.sources.syslog.host = 172.17.1.92
syslog-agent.sources.syslog.channels = rmc

# Configure sinks
syslog-agent.sinks.flat.type = FILE_ROLL
syslog-agent.sinks.flat.directory = /Users/ykim/Desktop/test
syslog-agent.sinks.flat.rollInterval = 0
syslog-agent.sinks.flat.channels = rmc
After a bit of wrangling with rsyslog configurations, it appeared as if rsyslog wasn't sending any packets. However, I decided to try out flume-og with a simple dump syslogTcp, and it correctly reads the syslog. At this point, I'm a bit lost and not sure what else to look at. Hopefully, someone on here can shed some light on what could be wrong.

Thanks,
Young Kim

HI,

try these for syslog tests:
echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
nc -v FLUME SYSLOG AGENT 5140 < /tmp/foo

> syslog-agent.sources.syslog.host = 172.17.1.92

Means, will only listen only on this interface, is the IP correct?

cheers,
 Alex

On Jul 10, 2012, at 10:59 PM, Young Kim wrote:

> Hey guys,
>
> I've been trying to use flume-ng (build in trunk) to read syslogs through tcp then write to a flat file, as seen in this configuration:
>
> # Source, sink, and channel name below
> syslog-agent.channels = rmc
> syslog-agent.sources = syslog
> syslog-agent.sinks = flat
>
> # Configure channels
> syslog-agent.channels.rmc.type = memory
>
> # Configure sources
> syslog-agent.sources.syslog.type = syslogTcp
> syslog-agent.sources.syslog.port = 5140
> syslog-agent.sources.syslog.host = 172.17.1.92
> syslog-agent.sources.syslog.channels = rmc
>
> # Configure sinks
> syslog-agent.sinks.flat.type = FILE_ROLL
> syslog-agent.sinks.flat.directory = /Users/ykim/Desktop/test
> syslog-agent.sinks.flat.rollInterval = 0
> syslog-agent.sinks.flat.channels = rmc
>
>
> After a bit of wrangling with rsyslog configurations, it appeared as if rsyslog wasn't sending any packets. However, I decided to try out flume-og with a simple dump syslogTcp, and it correctly reads the syslog. At this point, I'm a bit lost and not sure what else to look at. Hopefully, someone on here can shed some light on what could be wrong.
>
> Thanks,
> Young Kim
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Hey Alex,

The IP is correct, but I've altered it to say localhost. As for the syslog tests, I've ran it verbatim, but it doesn't register. However, when I alter /tmp/foo to include two new lines, flume registers it as a partial event.

It still seems odd to me that flume-ng isn't detecting the syslogtcp whereas flume-og is doing it perfectly. Is it possible that there is a bug in the way flume-ng is implementing it?

Thanks,
Young
On Tuesday, July 10, 2012 at 11:22 PM, alo alt wrote:

> HI,
>
> try these for syslog tests:
> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
> nc -v FLUME SYSLOG AGENT 5140 < /tmp/foo
>
> > syslog-agent.sources.syslog.host = 172.17.1.92
>
> Means, will only listen only on this interface, is the IP correct?
>
> cheers,
> Alex
>
> On Jul 10, 2012, at 10:59 PM, Young Kim wrote:
>
> > Hey guys,
> >
> > I've been trying to use flume-ng (build in trunk) to read syslogs through tcp then write to a flat file, as seen in this configuration:
> >
> > # Source, sink, and channel name below
> > syslog-agent.channels = rmc
> > syslog-agent.sources = syslog
> > syslog-agent.sinks = flat
> >
> > # Configure channels
> > syslog-agent.channels.rmc.type = memory
> >
> > # Configure sources
> > syslog-agent.sources.syslog.type = syslogTcp
> > syslog-agent.sources.syslog.port = 5140
> > syslog-agent.sources.syslog.host = 172.17.1.92
> > syslog-agent.sources.syslog.channels = rmc
> >
> > # Configure sinks
> > syslog-agent.sinks.flat.type = FILE_ROLL
> > syslog-agent.sinks.flat.directory = /Users/ykim/Desktop/test
> > syslog-agent.sinks.flat.rollInterval = 0
> > syslog-agent.sinks.flat.channels = rmc
> >
> >
> > After a bit of wrangling with rsyslog configurations, it appeared as if rsyslog wasn't sending any packets. However, I decided to try out flume-og with a simple dump syslogTcp, and it correctly reads the syslog. At this point, I'm a bit lost and not sure what else to look at. Hopefully, someone on here can shed some light on what could be wrong.
> >
> > Thanks,
> > Young Kim
> >
>
>
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>

After some debugging, I found out that the syslogTcp is working as intended. I didn't realize that debug no longer shows the resulting syslog messages on success. On that note, this may be a bit off-topic, but is there documentation showing how file_roll should be used to write to local disk?

Thanks,
Young

On Wednesday, July 11, 2012 at 11:12 AM, Young Kim wrote:

> Hey Alex,
>
> The IP is correct, but I've altered it to say localhost. As for the syslog tests, I've ran it verbatim, but it doesn't register. However, when I alter /tmp/foo to include two new lines, flume registers it as a partial event.
>
> It still seems odd to me that flume-ng isn't detecting the syslogtcp whereas flume-og is doing it perfectly. Is it possible that there is a bug in the way flume-ng is implementing it?
>
> Thanks,
> Young
>
>
> On Tuesday, July 10, 2012 at 11:22 PM, alo alt wrote:
>
> > HI,
> >
> > try these for syslog tests:
> > echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
> > nc -v FLUME SYSLOG AGENT 5140 < /tmp/foo
> >
> > > syslog-agent.sources.syslog.host = 172.17.1.92
> >
> > Means, will only listen only on this interface, is the IP correct?
> >
> > cheers,
> > Alex
> >
> > On Jul 10, 2012, at 10:59 PM, Young Kim wrote:
> >
> > > Hey guys,
> > >
> > > I've been trying to use flume-ng (build in trunk) to read syslogs through tcp then write to a flat file, as seen in this configuration:
> > >
> > > # Source, sink, and channel name below
> > > syslog-agent.channels = rmc
> > > syslog-agent.sources = syslog
> > > syslog-agent.sinks = flat
> > >
> > > # Configure channels
> > > syslog-agent.channels.rmc.type = memory
> > >
> > > # Configure sources
> > > syslog-agent.sources.syslog.type = syslogTcp
> > > syslog-agent.sources.syslog.port = 5140
> > > syslog-agent.sources.syslog.host = 172.17.1.92
> > > syslog-agent.sources.syslog.channels = rmc
> > >
> > > # Configure sinks
> > > syslog-agent.sinks.flat.type = FILE_ROLL
> > > syslog-agent.sinks.flat.directory = /Users/ykim/Desktop/test
> > > syslog-agent.sinks.flat.rollInterval = 0
> > > syslog-agent.sinks.flat.channels = rmc
> > >
> > >
> > > After a bit of wrangling with rsyslog configurations, it appeared as if rsyslog wasn't sending any packets. However, I decided to try out flume-og with a simple dump syslogTcp, and it correctly reads the syslog. At this point, I'm a bit lost and not sure what else to look at. Hopefully, someone on here can shed some light on what could be wrong.
> > >
> > > Thanks,
> > > Young Kim
> > >
> >
> >
> >
> > --
> > Alexander Alten-Lorenz
> > http://mapredit.blogspot.com
> > German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> >
> >
> >
>
>

Hi Young,
we've significantly improved user guide recently - you can get updated version either in repository (trunk) or in release 1.2.0 that is currently under vote (not released yet). Example of file_roll usage is present there.

Jarcec

On Jul 12, 2012, at 7:38 AM, Young Kim wrote:

> After some debugging, I found out that the syslogTcp is working as intended. I didn't realize that debug no longer shows the resulting syslog messages on success. On that note, this may be a bit off-topic, but is there documentation showing how file_roll should be used to write to local disk?
>
> Thanks,
> Young
> On Wednesday, July 11, 2012 at 11:12 AM, Young Kim wrote:
>
>> Hey Alex,
>>
>> The IP is correct, but I've altered it to say localhost. As for the syslog tests, I've ran it verbatim, but it doesn't register. However, when I alter /tmp/foo to include two new lines, flume registers it as a partial event.
>>
>> It still seems odd to me that flume-ng isn't detecting the syslogtcp whereas flume-og is doing it perfectly. Is it possible that there is a bug in the way flume-ng is implementing it?
>>
>> Thanks,
>> Young
>>
>> On Tuesday, July 10, 2012 at 11:22 PM, alo alt wrote:
>>
>>> HI,
>>>
>>> try these for syslog tests:
>>> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
>>> nc -v FLUME SYSLOG AGENT 5140 < /tmp/foo
>>>
>>>> syslog-agent.sources.syslog.host = 172.17.1.92
>>>
>>> Means, will only listen only on this interface, is the IP correct?
>>>
>>> cheers,
>>> Alex
>>>
>>> On Jul 10, 2012, at 10:59 PM, Young Kim wrote:
>>>
>>>> Hey guys,
>>>>
>>>> I've been trying to use flume-ng (build in trunk) to read syslogs through tcp then write to a flat file, as seen in this configuration:
>>>>
>>>> # Source, sink, and channel name below
>>>> syslog-agent.channels = rmc
>>>> syslog-agent.sources = syslog
>>>> syslog-agent.sinks = flat
>>>>
>>>> # Configure channels
>>>> syslog-agent.channels.rmc.type = memory
>>>>
>>>> # Configure sources
>>>> syslog-agent.sources.syslog.type = syslogTcp
>>>> syslog-agent.sources.syslog.port = 5140
>>>> syslog-agent.sources.syslog.host = 172.17.1.92
>>>> syslog-agent.sources.syslog.channels = rmc
>>>>
>>>> # Configure sinks
>>>> syslog-agent.sinks.flat.type = FILE_ROLL
>>>> syslog-agent.sinks.flat.directory = /Users/ykim/Desktop/test
>>>> syslog-agent.sinks.flat.rollInterval = 0
>>>> syslog-agent.sinks.flat.channels = rmc
>>>>
>>>>
>>>> After a bit of wrangling with rsyslog configurations, it appeared as if rsyslog wasn't sending any packets. However, I decided to try out flume-og with a simple dump syslogTcp, and it correctly reads the syslog. At this point, I'm a bit lost and not sure what else to look at. Hopefully, someone on here can shed some light on what could be wrong.
>>>>
>>>> Thanks,
>>>> Young Kim
>>>
>>>
>>> --
>>> Alexander Alten-Lorenz
>>> http://mapredit.blogspot.com
>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>

file_roll is detailed in the Flume user guide. It's generated in
target/docs when you do a full build. I've put it up at
http://people.apache.org/~juhanic/flume-docs/FlumeUserGuide.html#file-roll-sink
<http://people.apache.org/%7Ejuhanic/flume-docs/FlumeUserGuide.html#file-roll-sink>
for convenience, but the version may not match yours.

If the docs are insufficient let us know and we'll try to address it.

On 07/12/2012 02:38 PM, Young Kim wrote:
> After some debugging, I found out that the syslogTcp is working as
> intended. I didn't realize that debug no longer shows the resulting
> syslog messages on success. On that note, this may be a bit off-topic,
> but is there documentation showing how file_roll should be used to
> write to local disk?
>
> Thanks,
> Young
>
> On Wednesday, July 11, 2012 at 11:12 AM, Young Kim wrote:
>
>> Hey Alex,
>>
>> The IP is correct, but I've altered it to say localhost. As for the
>> syslog tests, I've ran it verbatim, but it doesn't register. However,
>> when I alter /tmp/foo to include two new lines, flume registers it as
>> a partial event.
>>
>> It still seems odd to me that flume-ng isn't detecting the syslogtcp
>> whereas flume-og is doing it perfectly. Is it possible that there is
>> a bug in the way flume-ng is implementing it?
>>
>> Thanks,
>> Young
>>
>> On Tuesday, July 10, 2012 at 11:22 PM, alo alt wrote:
>>
>>> HI,
>>>
>>> try these for syslog tests:
>>> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
>>> nc -v FLUME SYSLOG AGENT 5140 < /tmp/foo
>>>
>>>> syslog-agent.sources.syslog.host = 172.17.1.92
>>>
>>> Means, will only listen only on this interface, is the IP correct?
>>>
>>> cheers,
>>> Alex
>>>
>>> On Jul 10, 2012, at 10:59 PM, Young Kim wrote:
>>>
>>>> Hey guys,
>>>>
>>>> I've been trying to use flume-ng (build in trunk) to read syslogs
>>>> through tcp then write to a flat file, as seen in this configuration:
>>>>
>>>> # Source, sink, and channel name below
>>>> syslog-agent.channels = rmc
>>>> syslog-agent.sources = syslog
>>>> syslog-agent.sinks = flat
>>>>
>>>> # Configure channels
>>>> syslog-agent.channels.rmc.type = memory
>>>>
>>>> # Configure sources
>>>> syslog-agent.sources.syslog.type = syslogTcp
>>>> syslog-agent.sources.syslog.port = 5140
>>>> syslog-agent.sources.syslog.host = 172.17.1.92
>>>> syslog-agent.sources.syslog.channels = rmc
>>>>
>>>> # Configure sinks
>>>> syslog-agent.sinks.flat.type = FILE_ROLL
>>>> syslog-agent.sinks.flat.directory = /Users/ykim/Desktop/test
>>>> syslog-agent.sinks.flat.rollInterval = 0
>>>> syslog-agent.sinks.flat.channels = rmc
>>>>
>>>>
>>>> After a bit of wrangling with rsyslog configurations, it appeared
>>>> as if rsyslog wasn't sending any packets. However, I decided to try
>>>> out flume-og with a simple dump syslogTcp, and it correctly reads
>>>> the syslog. At this point, I'm a bit lost and not sure what else to
>>>> look at. Hopefully, someone on here can shed some light on what
>>>> could be wrong.
>>>>
>>>> Thanks,
>>>> Young Kim
>>>
>>>
>>> --
>>> Alexander Alten-Lorenz
>>> http://mapredit.blogspot.com
>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>

That does not look quite right.

'sink.directory' is not bold, which suggests that it is optional. But
there is no default. So where do the files go if you do not specify it?

On Thu, 2012-07-12 at 15:15 +0900, Juhani Connolly wrote:
> file_roll is detailed in the Flume user guide. It's generated in
> target/docs when you do a full build. I've put it up at
> http://people.apache.org/~juhanic/flume-docs/FlumeUserGuide.html#file-roll-sink for convenience, but the version may not match yours.
>
> If the docs are insufficient let us know and we'll try to address it.
>
> On 07/12/2012 02:38 PM, Young Kim wrote:
>
> > After some debugging, I found out that the syslogTcp is working as
> > intended. I didn't realize that debug no longer shows the resulting
> > syslog messages on success. On that note, this may be a bit
> > off-topic, but is there documentation showing how file_roll should
> > be used to write to local disk?
> >
> >
> > Thanks,
> > Young
> > On Wednesday, July 11, 2012 at 11:12 AM, Young Kim wrote:
> >
> > > Hey Alex,
> > >
> > >
> > > The IP is correct, but I've altered it to say localhost. As for
> > > the syslog tests, I've ran it verbatim, but it doesn't register.
> > > However, when I alter /tmp/foo to include two new lines, flume
> > > registers it as a partial event.
> > >
> > >
> > > It still seems odd to me that flume-ng isn't detecting the
> > > syslogtcp whereas flume-og is doing it perfectly. Is it possible
> > > that there is a bug in the way flume-ng is implementing it?
> > >
> > >
> > > Thanks,
> > > Young
> > >
> > >
> > > On Tuesday, July 10, 2012 at 11:22 PM, alo alt wrote:
> > >
> > > > HI,
> > > >
> > > >
> > > > try these for syslog tests:
> > > > echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message"
> > > > > /tmp/foo
> > > > nc -v FLUME SYSLOG AGENT 5140 < /tmp/foo
> > > >
> > > >
> > > > > syslog-agent.sources.syslog.host = 172.17.1.92
> > > >
> > > >
> > > > Means, will only listen only on this interface, is the IP
> > > > correct?
> > > >
> > > >
> > > > cheers,
> > > > Alex
> > > >
> > > >
> > > > On Jul 10, 2012, at 10:59 PM, Young Kim wrote:
> > > >
> > > >
> > > > > Hey guys,
> > > > >
> > > > >
> > > > > I've been trying to use flume-ng (build in trunk) to read
> > > > > syslogs through tcp then write to a flat file, as seen in this
> > > > > configuration:
> > > > >
> > > > >
> > > > > # Source, sink, and channel name below
> > > > > syslog-agent.channels = rmc
> > > > > syslog-agent.sources = syslog
> > > > > syslog-agent.sinks = flat
> > > > >
> > > > >
> > > > > # Configure channels
> > > > > syslog-agent.channels.rmc.type = memory
> > > > >
> > > > >
> > > > > # Configure sources
> > > > > syslog-agent.sources.syslog.type = syslogTcp
> > > > > syslog-agent.sources.syslog.port = 5140
> > > > > syslog-agent.sources.syslog.host = 172.17.1.92
> > > > > syslog-agent.sources.syslog.channels = rmc
> > > > >
> > > > >
> > > > > # Configure sinks
> > > > > syslog-agent.sinks.flat.type = FILE_ROLL
> > > > > syslog-agent.sinks.flat.directory = /Users/ykim/Desktop/test
> > > > > syslog-agent.sinks.flat.rollInterval = 0
> > > > > syslog-agent.sinks.flat.channels = rmc
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > After a bit of wrangling with rsyslog configurations, it
> > > > > appeared as if rsyslog wasn't sending any packets. However, I
> > > > > decided to try out flume-og with a simple dump syslogTcp, and
> > > > > it correctly reads the syslog. At this point, I'm a bit lost
> > > > > and not sure what else to look at. Hopefully, someone on here
> > > > > can shed some light on what could be wrong.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Young Kim
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Alexander Alten-Lorenz
> > > > http://mapredit.blogspot.com
> > > > German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> > >
> > >
> >
> >
>
>

You are right. It is an error. It should actually be in bold. Please feel free to submit a patch :-)

--
Hari Shreedharan
On Wednesday, July 11, 2012 at 11:21 PM, Stern, Mark wrote:

> That does not look quite right.
>
> 'sink.directory' is not bold, which suggests that it is optional. But
> there is no default. So where do the files go if you do not specify it?
>
> On Thu, 2012-07-12 at 15:15 +0900, Juhani Connolly wrote:
> > file_roll is detailed in the Flume user guide. It's generated in
> > target/docs when you do a full build. I've put it up at
> > http://people.apache.org/~juhanic/flume-docs/FlumeUserGuide.html#file-roll-sink for convenience, but the version may not match yours.
> >
> > If the docs are insufficient let us know and we'll try to address it.
> >
> > On 07/12/2012 02:38 PM, Young Kim wrote:
> >
> > > After some debugging, I found out that the syslogTcp is working as
> > > intended. I didn't realize that debug no longer shows the resulting
> > > syslog messages on success. On that note, this may be a bit
> > > off-topic, but is there documentation showing how file_roll should
> > > be used to write to local disk?
> > >
> > >
> > > Thanks,
> > > Young
> > > On Wednesday, July 11, 2012 at 11:12 AM, Young Kim wrote:
> > >
> > > > Hey Alex,
> > > >
> > > >
> > > > The IP is correct, but I've altered it to say localhost. As for
> > > > the syslog tests, I've ran it verbatim, but it doesn't register.
> > > > However, when I alter /tmp/foo to include two new lines, flume
> > > > registers it as a partial event.
> > > >
> > > >
> > > > It still seems odd to me that flume-ng isn't detecting the
> > > > syslogtcp whereas flume-og is doing it perfectly. Is it possible
> > > > that there is a bug in the way flume-ng is implementing it?
> > > >
> > > >
> > > > Thanks,
> > > > Young
> > > >
> > > >
> > > > On Tuesday, July 10, 2012 at 11:22 PM, alo alt wrote:
> > > >
> > > > > HI,
> > > > >
> > > > >
> > > > > try these for syslog tests:
> > > > > echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message"
> > > > > > /tmp/foo
> > > > >
> > > > > nc -v FLUME SYSLOG AGENT 5140 < /tmp/foo
> > > > >
> > > > >
> > > > > > syslog-agent.sources.syslog.host = 172.17.1.92
> > > > >
> > > > >
> > > > > Means, will only listen only on this interface, is the IP
> > > > > correct?
> > > > >
> > > > >
> > > > > cheers,
> > > > > Alex
> > > > >
> > > > >
> > > > > On Jul 10, 2012, at 10:59 PM, Young Kim wrote:
> > > > >
> > > > >
> > > > > > Hey guys,
> > > > > >
> > > > > >
> > > > > > I've been trying to use flume-ng (build in trunk) to read
> > > > > > syslogs through tcp then write to a flat file, as seen in this
> > > > > > configuration:
> > > > > >
> > > > > >
> > > > > > # Source, sink, and channel name below
> > > > > > syslog-agent.channels = rmc
> > > > > > syslog-agent.sources = syslog
> > > > > > syslog-agent.sinks = flat
> > > > > >
> > > > > >
> > > > > > # Configure channels
> > > > > > syslog-agent.channels.rmc.type = memory
> > > > > >
> > > > > >
> > > > > > # Configure sources
> > > > > > syslog-agent.sources.syslog.type = syslogTcp
> > > > > > syslog-agent.sources.syslog.port = 5140
> > > > > > syslog-agent.sources.syslog.host = 172.17.1.92
> > > > > > syslog-agent.sources.syslog.channels = rmc
> > > > > >
> > > > > >
> > > > > > # Configure sinks
> > > > > > syslog-agent.sinks.flat.type = FILE_ROLL
> > > > > > syslog-agent.sinks.flat.directory = /Users/ykim/Desktop/test
> > > > > > syslog-agent.sinks.flat.rollInterval = 0
> > > > > > syslog-agent.sinks.flat.channels = rmc
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > After a bit of wrangling with rsyslog configurations, it
> > > > > > appeared as if rsyslog wasn't sending any packets. However, I
> > > > > > decided to try out flume-og with a simple dump syslogTcp, and
> > > > > > it correctly reads the syslog. At this point, I'm a bit lost
> > > > > > and not sure what else to look at. Hopefully, someone on here