Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> scale of File Tail


Copy link to this message
-
Re: scale of File Tail
Huge file is better for file tailing adaptor actually.  FileTailing
adaptor checks every 2 seconds to see if file has rotated.  If the
rotating frequency is high, like rotating every seconds.  Then there
may be a chance of losing track of correct offset in the files.
Rotated log every couple minutes with large size, helps FileTailing
adaptor to track offset in the files correctly.
As far as throughput, it can stream up to 10MB/sec from our performance test.

regards,
Eric

On Thu, Dec 22, 2011 at 6:44 PM, AD <[EMAIL PROTECTED]> wrote:
> hey,
>
>  does anyone know if there are any throughput issues with FileTailing
> adaptors?   If i am writing huge files is there risk of missing lines
> (especially if the files are getting rotated).  I am just trying to decide
> if running map/reduce on a rotated log every X minutes is more safe than
> FileTail if the log writes get massively high (thousands/sec)
>
> Cheers,
> AD