Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Re: Flume tail


Copy link to this message
-
Re: Flume tail
Hi,

There is nothing CDH specific in this discussion so let's move to the
upstream user list which can be subscribed to here:

http://flume.apache.org/mailinglists.html

When appending just a single like you are likely hitting the default batch
size, which is 20. Try appending more than 20 lines or changing the batch
size to 1:

http://flume.apache.org/FlumeUserGuide.html#exec-source

Brock

On Mon, Nov 26, 2012 at 9:39 AM, Benc <[EMAIL PROTECTED]> wrote:

> Well nothing. If I start it up it all looks good. If I do
>
> echo test >> /tmp/ben.log changing the config to read that. I don't see
> anything in the log.
>
> I just see this in the flume.log
>
> 26 Nov 2012 15:37:39,263 INFO  [lifecycleSupervisor-1-1]
> (org.apache.flume.source.ExecSource.start:155)  - Exec source starting with
> command:tail -f /tmp/ben.log
>
> if for instance I change the command to "cat /etc/hosts"
>
> Then I see data in the log
>
> 26 Nov 2012 15:38:47,628 INFO  [lifecycleSupervisor-1-2]
> (org.apache.flume.source.ExecSource.start:155)  - Exec source starting with
> command:cat /etc/hosts
> 26 Nov 2012 15:38:47,645 INFO  [pool-4-thread-1]
> (org.apache.flume.source.ExecSource$ExecRunnable.run:307)  - Command [cat
> /etc/hosts] exited with 0
> 26 Nov 2012 15:38:47,648 INFO
>  [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.LoggerSink.process:70)  - Event: { headers:{} body:
> 23 23                                           ## }
> 26 Nov 2012 15:38:47,648 INFO
>  [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.LoggerSink.process:70)  - Event: { headers:{} body:
> 23 20 48 6F 73 74 20 44 61 74 61 62 61 73 65    # Host Database }
> 26 Nov 2012 15:38:47,648 INFO
>  [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.LoggerSink.process:70)  - Event: { headers:{} body:
> 23                                              # }
>
>
>
>
>
>
> On Monday, November 26, 2012 3:30:18 PM UTC, Brock Noland wrote:
>
>> Hi,
>>
>> What error or behavior are you seeing that you do not expect?
>>
>> Also, as someone else said, I would replace -f with -F.
>>
>> Brock
>>
>>
>> On Mon, Nov 26, 2012 at 9:27 AM, Benc <[EMAIL PROTECTED]> wrote:
>>
>>> Here is my config
>>>
>>> agent.sources = tail
>>>
>>> agent.channels = memoryChannel
>>>
>>> agent.sinks = loggerSink
>>>
>>>
>>> agent.sources.tail.type = exec
>>>
>>> agent.sources.tail.command = tail -f /Users/pj/work/flume_test.log
>>>
>>> agent.sources.tail.channels = memoryChannel
>>>
>>>
>>> agent.sinks.loggerSink.channel = memoryChannel
>>>
>>> agent.sinks.loggerSink.type = logger
>>>
>>>
>>> agent.channels.memoryChannel.**type = memory
>>>
>>> agent.channels.memoryChannel.**capacity = 100
>>>
>>> On Monday, November 26, 2012 3:26:55 PM UTC, Brock Noland wrote:
>>>
>>>> Hi,
>>>>
>>>> Can you share your configuration? Perhaps post it to pastbin?
>>>>
>>>> Brock
>>>>
>>>>
>>>> On Tue, Nov 20, 2012 at 11:10 AM, Benc <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi James
>>>>>
>>>>> I would like to get all of our application log files into HDFS is the
>>>>> high level overview, so I cam monitor for any errors. I did see that you
>>>>> can use the exec command, but when migrating up from flume to flumeNG tail
>>>>> does not work, or my configuration does not seem to work using exec = tail
>>>>> -f log.name
>>>>>
>>>>> I know that tail is not great as in the event of a failure it does not
>>>>> pick up from where it last read.
>>>>>
>>>>>
>>>>> On Tuesday, November 20, 2012 4:50:38 PM UTC, James Hogarth wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tuesday, 20 November 2012 16:38:53 UTC, Benc wrote:
>>>>>>>
>>>>>>> All with the new FlumeNG has anyone built a log reader that would
>>>>>>> replace tail?
>>>>>>
>>>>>>
>>>>>> Can you describe your use case a little more?
>>>>>>
>>>>>> The documentation for FlumeNG talks a bit about this and how the exec
>>>>>> source can be used with tail -F
>>>>>>
>>>>>> James
>>>>>>
>>>>>>  --
>>>>>
>>>>>
>>>>>
>>>>>
>>>>

Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB