Wang, Yongkun | Yongkun |... 2013-08-19, 02:29
-Re: sleep() in script doesn't work when called by exec Source
Brock Noland 2013-08-19, 13:08
In your case I would look at the spooling directory source.
On Sun, Aug 18, 2013 at 9:29 PM, Wang, Yongkun | Yongkun | BDD <
[EMAIL PROTECTED]> wrote:
> I am testing with apache-flume-1.4.0-bin.
> I made a naive python script for exec source to do throttling by calling
> sleep() function.
> But the sleep() doesn't work when called by exec source.
> Any ideas about this or do you have some simply solution for throttling
> instead of a custom source?
> Flume config:
> agent.sources = src1
> agent.sources.src1.type = exec
> agent.sources.src1.command = read-file-throttle.py
> import time
> with open("apache.log") as infile:
> for line in infile:
> line = line.strip()
> print line
> count += 1
> if count % 50000 == 0:
> now_time = time.time()
> diff = now_time - pre_time
> if diff < 10:
> #print "sleeping %s seconds ..." % (diff)
> pre_time = now_time
> Thank you very much.
> Best Regards,
> Yongkun Wang
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Wang, Yongkun | Yongkun |... 2013-08-20, 08:43
Brock Noland 2013-08-20, 14:58
Paul Chavez 2013-08-20, 16:15
Wang, Yongkun | Yongkun |... 2013-08-23, 05:26
Paul Chavez 2013-08-23, 21:26
Paul Chavez 2013-08-19, 17:56
Wang, Yongkun | Yongkun |... 2013-08-20, 08:44