- 1 EclipseLink Application Management, Analytics, Diagnostics and Performance
- 1.1 Document History
- 1.2 Problem Statement
- 1.3 Scope
- 1.4 Requirements
- 1.5 Analysis
- 1.5.1 Current EclipseLink PerformanceProfiler
- 1.5.2 Current EclipseLink Runtime MBean
- 1.5.3 Current JRockit JRMC Integration Points
- 18.104.22.168 Environment
- 22.214.171.124 JRockit Real Time
- 126.96.36.199 JRMC EclipseLink MBean method and attribute usage
- 188.8.131.52 JRMC EclipseLink MBean Trigger Rules
- 184.108.40.206 JRMC Live Method Profiler for EclipseLink User and API classes
- 220.127.116.11 JRMC Live Exception Count for EclipseLink API and JPA Spec Exceptions
- 18.104.22.168 JRMC Diagnostic Commands for running EclipseLink user or System Threads
- 22.214.171.124 JRMC Processor Usage Tracking for EclipseLink MBean Attributes
- 126.96.36.199 JRMC Live System Statistics tracking for EclipseLink MBean Attributes
- 188.8.131.52 JRMC Garbage Collection Visualization
- 184.108.40.206 JRMC Runtime Analyzer
- 220.127.116.11 JRMC Memory Leak Detector
- 18.104.22.168 JRMCD.exe
- 1.5.4 IBM J9 Integration
- 1.6 Third Party Tools
- 1.7 Design
- 1.8 Implementation
- 1.9 Testing
- 1.10 References
- 1.11 Competitive Intelligence
EclipseLink Application Management, Analytics, Diagnostics and Performance
|Date||Author||Version Description & Notes|
|20110214||Michael O'Brien||1.0 Initial draft started|
- This enhancement details work that will be done to enhance our basic JMX management API and add additional support for diagnostics and analytics reporting. We will also look at adding features that will enable developers to more accurately capture runtime performance and diagnostics information, as well as tools to interpret that data.
- There are two issues to this enhancement that separately deal with management and diagnostics reporting.
- Our existing runtime JMX management API that is exposed through the services MBean in any of the SUN JConsole, JRockit JRMC, Oracle JRF Enterprise Manager or JBoss web based console is sufficient but needs to be enhanced.
- Our existing runtime performance monitoring API could be enhanced with more JVM reporting - ideally that would not itself add overhead. This type of monitoring is currently done with JRockit - however it does not specifically target the JPA runtime API.
- The scope of this issue is limited to the primary EE application servers that we currently support via JMX. These are WebLogic, WebSphere, GlassFish and JBoss. We may also extend support to SE applications and EJB 3.1 Lite containers running exclusively in the web container.
- Metadata mappings from Annotations and ORM XML
- JPA 2.0 Metamodel
Dependency Injection Reference
- Spring, Session, Message-Driven and Singleton beans that reference or have injections of EMFs or EMs
JDBC Database Configuration
Extensibility and Multitenancy
- Management of shared schemas
- Before we can decide on how and where we will enhance our management, analytics and diagnostics support we need to answer the following questions.
- Q1) What do we currently provide
- Q2) What are we going to with existing enhancment requests and how will we merge these with new requirements.
- Q3) What new features can we provide to leverage new or existing management, performance and analytics tools that ship with the latest application server releases
- Q4) What new features can we provide on top of Q1-Q3 that are specific to EclipseLink that go beyond what is currently available.
Existing Enhancement Requests
- 238412: JMX: Add J2SE MBean JUnit client test suite
- 252786: WLS: Add extended metrics functionality to WebLogic specific JMX MBeans
- 253017: JMX: Refactor: MBean impl/interface naming conventions change
- 253019: JMX: Refactor: Additional Human readable time format for TimeConnectionEstablished
- 253020: JMX: Refactor: getCacheTypeFor() for NoIdentityMap should return "NoIdentity" instead "None"
- 256943: JMX: Remove the development MBean from EclipseLink (keep RuntimeServices)
- 256945: JMX: Append JPA persistence unit name to MBean session name
- 261038: JMX: values of ProfilingType attr should correspond to types in org.eclipse.persistence.config.ProfilerType
- 263271: JMX: A new attribute for persistenceUnit name
- 263274: JMX: An attribute returning dataSource name.
- 265540: JMX: Add runtime services MBean monitoring to Java SE clients
- 266016: JMX: MBeans: Add session and metadata runtime modification capability to runtime services mbeans
- 317991: JMX: initializeIdentityMap(className) requires unset EMPTY_STRING input validation handler to avoid ClassNotFoundException
- Official high level requirements
- R1: Address critical open issues general to all application servers
- R2: Expand platforms that we support JMX management on
- R2.1: Expand support for JMX management of JPA based applications to SE clients
- R2.2: Expand support for JMX management of JPA based applications to JEE6 EJB3.1 container managed deployments
- R2.2: (WebLogic) Integration points for weblogic.management.runtime WLSRuntimeMBeans
- R3: Manageability Requirements
- R3.1: Make MBeans change session aware - may not be an issue unless we are classified as configuration MBeans
- R3.2: (WebLogic EM) Provide discovery MBean with EM target types
- R3.3: Identify Key metrics?
- R4: Performance Tracking Requirements
- R4.1: (WebLogic) JRockit Mission Control Integration
- R4.2: (WebLogic) JRMC Flight Recorder integration
- R4.3: (WebLogic) Oracle Diagnostics Logging (ODL) usage/integration
- R4.4: Expose Runtime monitoring through a new MBean or the new generic PerformanceProfiler MBean
- This MBean would be exposed in EM?
- R4.4: Multicore utilization or thread parallelism API
- R5: Performance Analytics Requirements
- R5.1: (WebLogic) JRMC Runtime Analyser integration - to interpret JRMC-FR results
- R5.2: EclipseLink Performance Profiler analysis tool?
- R6: Scripting Requirements
- R6.1: (WebLogic) WLST jython scripting required?
- R6.2: WSADMIN (python) and OJBST scripting required?
- R7: Security Requirements
- R7.1: Verify security manager support for MBeans in WebSphere and JBoss
- R7.2: Determine how JConsole local access can be secured (remote access is via password or port control)
- R8: Localization
- R8.1: Command labels and messages must be I18N enabled.
- R9: Cache Integration (EclipseLink, Coherence, ehCache, Terracotta)
- R9.1: Expose MBean spanning multiple servers (no L2 cache)?
- R9.2: Expose multiple MBeans from multiple L1 EclipseLink caches
- R9.2: Expose MBean spanning multiple L2 caches
- R10: Third Party Integration
- R10.1: Spring Hyperic integration
- R10.2: Spring Insight integration
- Introduce Admin Role MBeans
- Investigate Scripting support for configuration changes (WLS for WebLogic, Python for WebSphere, ? for JBoss, ? for GlassFish)
WebLogic Specific Requirements
- WebLogic Server has several monitoring API join points that we can use either at the WebLogic MBean level or through events exposed to JRMC for servers running on the JRockit JVM. We will need to work closely with the WLS team to finalize the interfaces between EclipseLink and WLS.
Generic Server Requirements
- Ideally, any new API should be available to all supported EE servers (not just WebLogic) and optionally SE clients.
- R: Expose PerformanceProfiler/PerformanceMonitor events through a new MBean generic to all EE servers
- non-WebLogic EE servers may not benefit from performance tracking provided by JRockit out of the box. These servers (WebSphere, JBoss and GlassFish) can however use the JRockit JVM with minimal configuration changes.
- Our analysis will take two phases.
- 1) What do we currently provide?
- Specifically from EclipseLink (like our PerformanceProfiler)
- Via our JMX MBean.
- Or generically via the SUN JVM bin tools
- Or specifically via the JRockit JVM and JRMC (JRockit Mission Control and Flight Recorder).
- 2) What can we enhance or expose through API?
- What runtime properties can be added to our JMX MBean.
- What runtime properties can be tracked via existing or new WLSRuntimeBeans.
- What other 3rd party performance tracking and analytic tools can be integrated with.
- 1) What do we currently provide?
Current EclipseLink PerformanceProfiler
Current EclipseLink Runtime MBean
Current JRockit JRMC Integration Points
- WebLogic 10.3.4.0 20110115 version
- Running JRockit (32-bit on 64-bit)
- Oracle JRockit(R) (build R28.1.1-14-139783-1.6.0_22-20101206-0241-windows-ia32, compiled mode)
- Running EclipseLink 2.3 trunk replaces 2.1.2 shipped with WebLogic 10.3.4
- EE application is the Collatz distributed EE application with 6 external JVM's running various IA64, IA32 and UltraSparc implementations.
JRockit Real Time
- [JRockit Profiling and Tuning http://download.oracle.com/docs/cd/E15289_01/doc.40/e15060/toc.htm]
JRMC EclipseLink MBean method and attribute usage
JRMC EclipseLink MBean Trigger Rules
- An example would be a trigger on Number of Objects In All Identity Maps. In my [distributed case study] after running the server for 24 hours we reach around 26000 objects in the identity map before the map is reset back to 6000. Therefore if we put a trigger at say 26000 we should see an event once a day on a load of 1 transaction per second.
JRMC Live Method Profiler for EclipseLink User and API classes
JRMC Live Exception Count for EclipseLink API and JPA Spec Exceptions
JRMC Diagnostic Commands for running EclipseLink user or System Threads
JRMC Processor Usage Tracking for EclipseLink MBean Attributes
JRMC Live System Statistics tracking for EclipseLink MBean Attributes
JRMC Garbage Collection Visualization
JRMC Runtime Analyzer
JRMC Memory Leak Detector
IBM J9 Integration
- For users of WebSphere 6,7 and 8
Third Party Tools
Eclipse Memory Analyzer Tool (MAT)
Eclipse Memory Analyzer Tool for Java Heap Analysis.
- Available options:
- Batch Reading
- Batch Writing
- Partial Attribute Query
- Parameterized SQL
- Bulk Update Query
- Transformation Mappings
- Order of SQL
- Avoiding instanceof checks
- Statement caches
- Leverage CallableStatement
- Cache Coordination (coordination)
- Lazy Loading - via proxy instrumentation
- Change Tracking - via proxy instrumentation
- Optimizing Joins
- L1 Cache parameters
- L2 Cache integration
- Sequence allocation (size)
- In memory database
- When to use Pessimistic Locking over Optimistic
- Patterns for 2-phase commit - XA
Asymmetric Data Access
- Most applications perform more reading that writing to the database. We therefore need to provide performance analysis tools that help to optimize the cache for the customer scenario.
- What exactly is the magic number of virtual machines that are needed for a particular application in order that a performance gain will happen if we switch to an L2 cache.
- Does warming of the cache improve read performance.