Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Re: Flume tail


Copy link to this message
-
Re: Flume tail
Brock Noland 2012-11-26, 15:47
Hi,

There is nothing CDH specific in this discussion so let's move to the
upstream user list which can be subscribed to here:

http://flume.apache.org/mailinglists.html

When appending just a single like you are likely hitting the default batch
size, which is 20. Try appending more than 20 lines or changing the batch
size to 1:

http://flume.apache.org/FlumeUserGuide.html#exec-source

Brock

On Mon, Nov 26, 2012 at 9:39 AM, Benc <[EMAIL PROTECTED]> wrote:

> Well nothing. If I start it up it all looks good. If I do
>
> echo test >> /tmp/ben.log changing the config to read that. I don't see
> anything in the log.
>
> I just see this in the flume.log
>
> 26 Nov 2012 15:37:39,263 INFO  [lifecycleSupervisor-1-1]
> (org.apache.flume.source.ExecSource.start:155)  - Exec source starting with
> command:tail -f /tmp/ben.log
>
> if for instance I change the command to "cat /etc/hosts"
>
> Then I see data in the log
>
> 26 Nov 2012 15:38:47,628 INFO  [lifecycleSupervisor-1-2]
> (org.apache.flume.source.ExecSource.start:155)  - Exec source starting with
> command:cat /etc/hosts
> 26 Nov 2012 15:38:47,645 INFO  [pool-4-thread-1]
> (org.apache.flume.source.ExecSource$ExecRunnable.run:307)  - Command [cat
> /etc/hosts] exited with 0
> 26 Nov 2012 15:38:47,648 INFO
>  [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.LoggerSink.process:70)  - Event: { headers:{} body:
> 23 23                                           ## }
> 26 Nov 2012 15:38:47,648 INFO
>  [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.LoggerSink.process:70)  - Event: { headers:{} body:
> 23 20 48 6F 73 74 20 44 61 74 61 62 61 73 65    # Host Database }
> 26 Nov 2012 15:38:47,648 INFO
>  [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.LoggerSink.process:70)  - Event: { headers:{} body:
> 23                                              # }
>
>
>
>
>
>
> On Monday, November 26, 2012 3:30:18 PM UTC, Brock Noland wrote:
>
>> Hi,
>>
>> What error or behavior are you seeing that you do not expect?
>>
>> Also, as someone else said, I would replace -f with -F.
>>
>> Brock
>>
>>
>> On Mon, Nov 26, 2012 at 9:27 AM, Benc <[EMAIL PROTECTED]> wrote:
>>
>>> Here is my config
>>>
>>> agent.sources = tail
>>>
>>> agent.channels = memoryChannel
>>>
>>> agent.sinks = loggerSink
>>>
>>>
>>> agent.sources.tail.type = exec
>>>
>>> agent.sources.tail.command = tail -f /Users/pj/work/flume_test.log
>>>
>>> agent.sources.tail.channels = memoryChannel
>>>
>>>
>>> agent.sinks.loggerSink.channel = memoryChannel
>>>
>>> agent.sinks.loggerSink.type = logger
>>>
>>>
>>> agent.channels.memoryChannel.**type = memory
>>>
>>> agent.channels.memoryChannel.**capacity = 100
>>>
>>> On Monday, November 26, 2012 3:26:55 PM UTC, Brock Noland wrote:
>>>
>>>> Hi,
>>>>
>>>> Can you share your configuration? Perhaps post it to pastbin?
>>>>
>>>> Brock
>>>>
>>>>
>>>> On Tue, Nov 20, 2012 at 11:10 AM, Benc <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi James
>>>>>
>>>>> I would like to get all of our application log files into HDFS is the
>>>>> high level overview, so I cam monitor for any errors. I did see that you
>>>>> can use the exec command, but when migrating up from flume to flumeNG tail
>>>>> does not work, or my configuration does not seem to work using exec = tail
>>>>> -f log.name
>>>>>
>>>>> I know that tail is not great as in the event of a failure it does not
>>>>> pick up from where it last read.
>>>>>
>>>>>
>>>>> On Tuesday, November 20, 2012 4:50:38 PM UTC, James Hogarth wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tuesday, 20 November 2012 16:38:53 UTC, Benc wrote:
>>>>>>>
>>>>>>> All with the new FlumeNG has anyone built a log reader that would
>>>>>>> replace tail?
>>>>>>
>>>>>>
>>>>>> Can you describe your use case a little more?
>>>>>>
>>>>>> The documentation for FlumeNG talks a bit about this and how the exec
>>>>>> source can be used with tail -F
>>>>>>
>>>>>> James
>>>>>>
>>>>>>  --
>>>>>
>>>>>
>>>>>
>>>>>
>>>>

Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/