Jump to: navigation, search

SMILA/Project Concepts/Concept for a Performance Measurement Framework

Goals of the Performance Measurement Framework:

Framework for:

  • Measurements of Application Metrics like
    • Response time
    • Throughput
    • Resource Utilization
    • Workload
  • A totally distributed application
  • A heterogeneous Components (Hardware / Operation System)
  • Applications are based on Java
  • Profiling (Monitoring on instruction level) is not important


Survey

The Concept for the Framework can be divided into two components:

  • Measurement Interface/API - Component: Measurements are taken from application\-specific results. Therefore the Applications have to measure the metric itself. The results can be delivered to the Measurement interface/API which can be used from the Data Collection Component.
  • Data Collection Component: The Data Collection Component can contact the Measurement Interface/API that is used in parts of the distributed application to collect the results of the measurement. Furthermore this component can analyze/convert the data and can create statistics or graphs.


Survey of techniques for the Measurement Interface/API:

  • Logging
    • Every Part of the distributed Application can use logging to save results of the measurement.
      • Advantages: Simple approach
      • Disadvantages: Synchronization of Measurement entries
      • Unsolved: the Collection of the logging files
        • Can be solved with an implementation over JMS /ActiveMQ
  • JMX (Java Management Extensions)
    • Framework for Management and Monitoring of Java\-Applications
    • Advantages:
      • Simple publication of application properties (can be measurement results)
        • MBeans ( like Pojo\-Objects with getter/setter)
        • Notifications possible
    • Disadvantages:
      • MBeans Properties has to be polled (probably, otherwise use of notifications)
  • ARM (Application Response Measurement)
    • Open standard ([[1]])
    • "for monitoring and diagnosing performance bottlenecks within complex enterprise applications that use loosely\-coupled designs or service\-oriented architectures"
    • Standard contains the API for C and java that allows timing information associated with each step in processing a transaction to be logged to a remote server for later analysis
    • Contains Functions for Performance Counters, Monitoring of Transactions, etc.
    • Implementations:
      • Open Source: open\-arm.sourceforge.net
        • Alpha/Beta' status (version 0.009)
        • Contains for the distribution of results:
          • Logging
          • SNMP
      • ARM\-SDK4 (distributed from the standardization group)
        • Uses the C Lib and JNI for the java API
        • Measurement functions are implement
        • Implementation of the Distribution of measurement results
          • ONLY: logging
      • Commercial: tang\-it.com
        • Commercial!
        • E.g.: Supports the collection/distribution of measurement data to a database server
        • Advantages: best solution for the Performance Measurement Framework
        • Disadvantages: there is no (open source, non-commercial) implementation that supports distributed applications and distributed measurements

nSurvey of techniques for the Data Collection Component:

Most Techniques are developed for the monitoring of J2EE\-Applications/Servers and for profiling Java\-Applications


  • JManage (jmanage.org): Client for JMX: runs in a Web server:

Can be used to access applications that use JMX Can create graphs from the returned data


  • Commercial Tools:
    • DynaTrace
    • Intrascope


  • Eclipse Test & Performance Tools Platform Project ([[2] www.eclipse.org/tptp]) :
  • TPTP provides powerful frameworks and services for an open platform upon which developers build unique test and performance tools
  • TPTP can be used for profiling of Java\-Applications and JMX\-Application can be managed:
  • Advantages:
    • Logging concept : TPTP supports CBE (Common Base Events). It can read logs and can convert them to CBE. CBE can be used for statistical analyze of the data
    • Open Platform: The TPTP\- Framework can be extended
      • Agent Controller specific:
        • E.g. Eclipse Help:
          • TPTP Data Collection Framework / How to write a TPTP Data Collection Agent
          • Extending the monitoring, profiling, and testing functions


  • Problems/Disadvantages:
    • JMX can used for managing JMX\-Applications, but JMX information cannot be used to creating graphs/statistical
    • For the "Statistical" Launch Configuration there is no Agent (Agent is a collecting service that can be started in the Agent Controller) in the Agent Controller that can use ARM or JMX - Clients (JMX only for specific JBoss/Jonas, no JSR\-160').
    • And only one agent can be selected in eclipse TPTP( but: according to the documentation it should be possible to use more agents [[3]] )
    • Therefore for the use of TPTP for this Framework there have to be some further development ( a special agent that can address JMX or ARM and simultaneously the windows PerfMon functionality)


  • Eclipse Cosmos:

The COSMOS (Community\-driven Systems Management in Open Source) project aims to provide an extensible, standards\-based framework upon which software developers can create specialized, differentiated and inter\-operable offerings of tools for system management.

    • Review: Cosmos is no option. Release V1.0 is Sommer 2008


Review:

  • the non-commercial arm - software allows no distributed application (only the logging facility is implemented, with open-arm snmp, probably this is a option!?)
  • JMX is used by many applications to allow the management of (runtime) properties, but it isnt often used for measurement functionality (but it can be used for it)

Further Steps:

  • Open Issue: How can JMX/ARM integrated into the TPTP-Framework to use the "statisticals" launch configuration (for Graphs)?
    • Own Implementation of an agent for the agent controller (work?)
    • Can this agent support JMX/ARM and Windows Performace Information?
  • Open Issue: Alternatives?