-Throughput issue/load balancing
Camp, Roy 2012-07-03, 03:18
We have a couple dozen agents connecting to two collectors and a majority of events appear to be flowing into one collector. I am trying to load archived data from two of the agents (two different hosts) but they hit a throughput limit. When investigating it appears that all the events are only flowing to one collector. Shouldn't flume be utilizing the second collector for increased throughput?
All agents: thriftSource(12345) | autoE2EChain;
Both collectors: autoCollectorSource | [ackChecker cassandraBasin("analyticsks", "127.0.0.1:9160"), < lazyOpen stubbornAppend thriftSink("127.0.0.1",30313) ? diskFailover insistentOpen lazyOpen stubbornAppend thriftSink("127.0.0.1",30313) >];
Agent logs show them opening a connection to both collectors upon startup.