Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: Flume-748. Metric collection infrastructure


Copy link to this message
-
Re: Review Request: Flume-748. Metric collection infrastructure
Arvind Prabhakar 2012-07-06, 02:23

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5782/
-----------------------------------------------------------

(Updated July 6, 2012, 2:23 a.m.)
Review request for Flume.
Changes
-------

Fixed a typo and a bug due to which the sink counter was not reporting the correct stop time for AvroSink.
Description
-------

This change introduces a simple, minimally intrusive, standard metric collection framework based on the modified concept of CounterGroups that are currently used within the system. Instead of modifying the CounterGroup construct, this change introduces the notion of a MonitoredCounterGroup that uses the component name and type to register Standard MBeans in the PlatformMBeanSever. These MBeans are never deregistered and the components are wired to overwrite the run state on every restart.

Specialized implementation of MonitoredCounterGroup for Source, Channel and Sinks are provided which expose standard metrics that will apply generally to all component implementations. Individual components can chose to populate the necessary metrics and ignore the ones that may not directly apply.
This addresses bug FLUME-748.
    https://issues.apache.org/jira/browse/FLUME-748
Diffs (updated)
-----

  /trunk/conf/flume-env.sh.template 1358021
  /trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 1358021
  /trunk/flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 1358021
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/ChannelCounter.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/ChannelCounterMBean.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/ChannelProcessorCounter.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/MonitoredCounterGroup.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/SinkCounter.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/SinkCounterMBean.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/SinkProcessorCounter.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/SourceCounter.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/SourceCounterMBean.java PRE-CREATION
  /trunk/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1358021
  /trunk/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1358021
  /trunk/flume-ng-core/src/test/java/org/apache/flume/instrumentation/MonitoredCounterGroupTest.java PRE-CREATION
  /trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 1358021
  /trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1358021
  /trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 1358021

Diff: https://reviews.apache.org/r/5782/diff/
Testing
-------

Updated the tests where necessary, added some tests to exercise the monitored counter groups and did manual testing using jconsole to ensure that the metrics were being published correctly.
Thanks,

Arvind Prabhakar