Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> sleep() in script doesn't work when called by exec Source


Copy link to this message
-
Re: sleep() in script doesn't work when called by exec Source
Hi,

Can you share the details of this?  It shouldn't die with large files.
On Tue, Aug 20, 2013 at 3:43 AM, Wang, Yongkun | Yongkun | BDD <
[EMAIL PROTECTED]> wrote:

>  Thanks Brock.
>
>  I tried spooling directory, if the file dropped in spoolDir was too
> large, flume also died. There should be a blocking.
> Will start a standalone script process to drop small files.
>
>  Best Regards,
> Yongkun Wang
>
>  On 2013/08/19, at 22:08, Brock Noland wrote:
>
>  In your case I would look at the spooling directory source.
>
>
> On Sun, Aug 18, 2013 at 9:29 PM, Wang, Yongkun | Yongkun | BDD <
> [EMAIL PROTECTED]> wrote:
>
>>  Hi,
>>
>>  I am testing with apache-flume-1.4.0-bin.
>> I made a naive python script for exec source to do throttling by calling
>> sleep() function.
>> But the sleep() doesn't work when called by exec source.
>> Any ideas about this or do you have some simply solution for throttling
>> instead of a custom source?
>>
>>  Flume config:
>>
>>
>> agent.sources = src1
>> agent.sources.src1.type = exec
>> agent.sources.src1.command = read-file-throttle.py
>>
>>
>>  read-file-throttle.py:
>>
>>
>> #!/usr/bin/python
>> import time
>>
>> count=0
>> pre_time=time.time()
>> with open("apache.log") as infile:
>>     for line in infile:
>>         line = line.strip()
>>         print line
>>         count += 1
>>         if count % 50000 == 0:
>>             now_time = time.time()
>>             diff = now_time - pre_time
>>             if diff < 10:
>>                 #print "sleeping %s seconds ..." % (diff)
>>                 time.sleep(diff)
>>                 pre_time = now_time
>>
>>
>>
>>  Thank you very much.
>>
>>  Best Regards,
>> Yongkun Wang
>>
>
>
>
>  --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
>
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB