Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> FLume OG Choke Limit Not Working


+
James Stewart 2013-03-11, 23:53
+
James Stewart 2013-03-14, 02:35
Copy link to this message
-
Re: FLume OG Choke Limit Not Working
Hi James,

According to source codes, depending on event body size, choke limit may
not work accurately.
One scenario I can imagine for this case is:

  * Given choke limit 10 which means 1 KB bucket size and event body
    size 100KB,
  * when append() is called in choke deco, it may wait for 2 times (200
    milliseconds) as 100KB is larger than bucket size (1 KB)
  * but at third time it just ships the event to Sink no matter how
    large the event body is,
  * this can happen about 3 times within a seconds (every 100 * 3
    milliseconds).
  * This may result in a spike of more than 300KB (100KB * 3)/sec;
    2.4Mbit/sec.

So I am curious about the size of your event body.

-JS

On 3/13/13 7:35 PM, James Stewart wrote:
>
> Hello again,
>
> Further to my previous problem with the Choke decorator under Flume
> OG, I've tried separating out my 'aggregator' into 3 logical nodes as
> follows...
>
> OLD:
>
> exec setChokeLimit aggregator.mydomain.com mychoke 150
>
> exec config aggregator.mydomain.com 'collectorSource(35853)'
> 'batch(100, 1000) gzip choke("mychoke")
> agentBESink("collector.mydomain.com", 35853)'
>
> NEW:
>
> exec setChokeLimit aggregator.mydomain.com mychoke 150
>
> exec config batcher 'collectorSource(35853)' 'batch(100, 1000)
> agentBESink("aggregator.mydomain.com", 35854)'
>
> exec config zipper 'collectorSource(35854)' 'gzip
> agentBESink("aggregator.mydomain.com ", 35855)'
>
> exec config choker 'collectorSource(35855)' 'choke("mychoke")
> agentBESink("collector.mydomain.com ", 35853)'
>
> Looking at the stats, it seems that the 'choker' decorator is having
> no effect on the data passing through it. Even when I set 'mychoke' to
> 10, (which equates to 80Kbit/sec) I still see regular spikes above
> 1.5Mbit/sec.
>
> The Flume User Guide says:
>
> /"The choke decorator works as follows: when append() is called on the
> sink to which the choke is attached, the append() call works normally
> if the amount of data transferred (during a small duration of time) is
> within the limit assigned to the choke-id corresponding to the choke.
> If the limit has been exceeded, then append() is blocked for a small
> duration of time."/
>
> Our traffic is quite bursty -- could this be a problem for the Choke
> decorator?
>
> Any help is much appreciated as I've hit a bit of a wall.
>
> Cheers,
>
> James
>
> *From:*James Stewart [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, 12 March 2013 10:54 AM
> *To:* [EMAIL PROTECTED]
> *Subject:* FLume OG Choke Limit Not Working
>
> Hello all,
>
> I'm using flume OG (unable to upgrade to NG at this stage) and I am
> having trouble with the choke decorator.
>
> I am aggregating the data flows from several logical nodes at a single
> 'aggregator' logical node. The data flows should be batched, zipped,
> choked and then sent on to another 'collector' logical node. I am
> using the following config to achieve this:
>
> exec setChokeLimit aggregator.mydomain.com mychoke 150
>
> exec config aggregator.mydomain.com 'collectorSource(35853)'
> 'batch(100, 1000) gzip choke("mychoke")
> agentBESink("collector.mydomain.com", 35853)'
>
> The choke decorator should limit transfer to 150KB/sec, which equates
> to 1.2Mbit/sec. However I am regularly recording Flume traffic spikes
> of 5Mbit/sec and more.
>
> Can anybody suggest what I might be doing wrong? Is it ok to chain the
> batch, gzip and choke decorators like this, or should they each be in
> a separate logical node?
>
> Thanks,
>
> James
>
> ------------------------------------------------------------------------
>
> Information contained in this communication (including any
> attachments) is confidential and may be privileged or subject to
> copyright. If you have received this communication in error you are
> not authorised to use the information in any way and Optiver requests
> that you notify the sender by return email, destroy all copies and
> delete the information from your system. Optiver does not represent,
Jeong-shik Jang / [EMAIL PROTECTED]
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting