Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Recommendation of parameters for better performance with File Channel


Copy link to this message
-
Recommendation of parameters for better performance with File Channel
Hi

I am able to write maximum 1.5 MB/sec data to HDFS (without compression)
using File Channel. Are there any recommendations to improve the
performance?
Has anybody achieved around 10 MB/sec with file channel ? If yes please
share the
configuration like (Hardware used, RAM allocated and batch sizes of
source,sink and channels).

Following are the configuration details :
=======================
I am using a machine with reasonable hardware configuration:
Quadcore 2.00 GHz processors and 4 GB RAM.

Command line options passed to flume agent :
-DJAVA_OPTS="-Xms1g -Xmx4g -Dcom.sun.management.jmxremote
-XX:MaxDirectMemorySize=2g"

Agent Configuration:
============agent.sources = avro-collection-source spooler
agent.channels = fileChannel
agent.sinks = hdfsSink fileSink

# For each one of the sources, the type is defined

agent.sources.spooler.type = spooldir
agent.sources.spooler.spoolDir =/root/test_data
agent.sources.spooler.batchSize = 1000
agent.sources.spooler.channels = fileChannel

# Each sink's type must be defined
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.hdfs.path=hdfs://mltest2001/flume/release3Test

agent.sinks.hdfsSink.hdfs.fileType =DataStream
agent.sinks.hdfsSink.hdfs.rollSize=0
agent.sinks.hdfsSink.hdfs.rollCount=0
agent.sinks.hdfsSink.hdfs.batchSize=1000
agent.sinks.hdfsSink.hdfs.rollInterval=60

agent.sinks.hdfsSink.channel= fileChannel

agent.channels.fileChannel.type=file
agent.channels.fileChannel.dataDirs=/root/flume_channel/dataDir13
agent.channels.fileChannel.checkpointDir=/root/flume_channel/checkpointDir13

Regards,
Jagadish
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB