Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request: Flume-748. Metric collection infrastructure


Copy link to this message
-
Review Request: Flume-748. Metric collection infrastructure

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5782/
-----------------------------------------------------------

Review request for Flume.
Description
-------

This change introduces a simple, minimally intrusive, standard metric collection framework based on the modified concept of CounterGroups that are currently used within the system. Instead of modifying the CounterGroup construct, this change introduces the notion of a MonitoredCounterGroup that uses the component name and type to register Standard MBeans in the PlatformMBeanSever. These MBeans are never deregistered and the components are wired to overwrite the run state on every restart.

Specialized implementation of MonitoredCounterGroup for Source, Channel and Sinks are provided which expose standard metrics that will apply generally to all component implementations. Individual components can chose to populate the necessary metrics and ignore the ones that may not directly apply.
This addresses bug FLUME-748.
    https://issues.apache.org/jira/browse/FLUME-748
Diffs
-----

  /trunk/conf/flume-env.sh.template 1356576
  /trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 1356576
  /trunk/flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 1356576
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/ChannelCounter.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/ChannelCounterMBean.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/ChannelProcessorCounter.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/MonitoredCounterGroup.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/SinkCounter.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/SinkCounterMBean.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/SinkProcessorCounter.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/SourceCounter.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/SourceCounterMBean.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1356576
  /trunk/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1356576
  /trunk/flume-ng-core/src/test/java/org/apache/flume/instrumentation/MonitoredCounterGroupTest.java PRE-CREATION
  /trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 1356576
  /trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1356576
  /trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 1356576

Diff: https://reviews.apache.org/r/5782/diff/
Testing
-------

Updated the tests where necessary, added some tests to exercise the monitored counter groups and did manual testing using jconsole to ensure that the metrics were being published correctly.
Thanks,

Arvind Prabhakar