Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> File channel configuration


Copy link to this message
-
File channel configuration
Hi

I'm trying to figure out the best way to configure the file channel for
maximum throughput and have a couple of questions.

1. What is the best hard disk layout? An ssd for the checkpoint directory
and a separate disk for each file channel on the agent?

2. Can multiple discs be utilized for a single channel? I could only seem
to configure a single data directory.

3. There is a comment in the documentation that mentions adding more sinks
to drain the channel faster. If my final agent sink was hdfs does that mean
configuring two hdfs sinks using a sink group to drain a single channel on
an agent? I noticed you can configure thread pools on the hdfs sink but
haven't looked into it.

4. Does it make sense to have my agent run two channels both with sinks
writing to a single hdfs cluster each configured with q separate data disk
and have the previous agent round robin deliver to it?

Thanks for any input anyone has
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB