Difference between revisions of "MemoryAnalyzer/Contributor Reference"

Latest revision as of 03:39, 11 December 2023

Getting Started

This page is meant to help you contribute to the Memory Analyzer project.

Workspace

Setup

Install Eclipse and configure it to develop Java 5 applications.

• You can use 3.x or 4.x as you prefer.

Setup your development environment:

• Via the Update Manager Help -> Install New Software... install:
  • Eclipse BIRT Framework
  • IBM Diagnostic Tool Framework for Java (See IBM Diagnostic Tool Framework for Java Version 1.12 . An Update Site is at https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/runtimes/tools/dtfj/ .) This is needed to compile and run with the DTFJ adapter which is part of Memory Analyzer and allows Memory Analyzer to read dumps from IBM virtual machines for Java.
• Configure the Code Formatter Template:
  • Preferences -> Java -> Code Style -> Formatter -> Import... and import this template.

Get the source

Since December 2014 the Memory Analyzer code is stored in a git repository. The URLs to access it are:

ssh://<username>@git.eclipse.org:29418/mat/org.eclipse.mat

https://<username>@git.eclipse.org/r/mat/org.eclipse.mat

A web browsable repository is available at https://git.eclipse.org/c/mat/org.eclipse.mat.git/

The Eclipse Memory Analyzer project also uses Gerrit, and the Gerrit code reviews are at: https://git.eclipse.org/r/#/q/mat/org.eclipse.mat

and the Gerrit version of the source repositories is available at:

ssh://git.eclipse.org:29418/mat/org.eclipse.mat
https://git.eclipse.org/r/mat/org.eclipse.mat.git

You can clone the repository using your favorite Git client. More information: Git, EGit/User Guide.

There are eclipse .project files, so that the projects (plugins, features, etc...) can be easily imported into the Eclipse IDE.

If you do not intend to build the MAT update site and standalone distributions, then it is enough to import only the MAT plugins.

If you want to run locally the Maven/Tycho build for MAT, which produces an update site and standalone RCP products, follow the instructions on the build Wiki page: MemoryAnalyzer/Building_MAT_With_Tycho

If you do not have BIRT installed then there will be compilation errors in the org.eclipse.mat.chart and org.eclipse.mat.chart.ui projects.

If you do not have the IBM DTFJ feature installed then there will be compilation errors in the org.eclipse.mat.dtfj project.

Configure API Tooling Baseline

In order to guarantee that no API breaking changes are introduced we recomment using the PDE API Tooling and defining the latest released version of MAT as an API Baseline. Here is a short description how this could be done:

• Download the latest released version in order to use it as an API Baseline
  • Go to the MAT download page
  • Download the "Archived Update Site" zip file for the latest release
  • Unzip the file somewhere locally
• Configure the API Baseline in the IDE
  • In the IDE open Window -> Preferences -> Plug-in Development -> API Baselines
  • Press Add Baseline
  • Select An Existing Eclipse installation Directory as the source for this baseline.
  • Browse and select as Location the directory in which the zip was extracted
  • Enter a name for the baseline, click Finish and confirm the rest of the dialogs

Once the API Tooling is properly setup, one will see errors reported if API changes are introduced.

Launch Configuration

Launch the Memory Analyzer as stand-alone RCP:

• Create a new Eclipse Application configuration
• Run a product: org.eclipse.mat.ui.rcp.MemoryAnalyzer
• Launch with: plug-ins selected below only
  • Deselect org.eclipse.mat.tests and org.eclipse.mat.ui.rcp.tests
  • Deselect Target Platform and click Add Required Plug-ins
  • Select org.eclipse.pde.runtime (3.3) or org.eclipse.ui.views.log (3.4 or later) to include the Error Log
  • Select com.ibm.dtfj.api com.ibm.dtfj.j9 com.ibm.dtfj.phd com.ibm.dtfj.sov if you have installed the IBM DTFJ feature and wish to process dumps from IBM virtual machines
  • Select com.ibm.java.doc.tools.dtfj for help for IBM DTFJ
  • Eclipse >= Neon: Select org.eclipse.equinox.ds and org.eclipse.equinox.event

or as feature plugged into the IDE:

• Create a new Eclipse Application configuration
• Run a product: org.eclipse.sdk.ide
• Launch with: plug-ins selected below only
  • Deselect org.eclipse.mat.tests and org.eclipse.mat.ui.rcp
  • Select com.ibm.dtfj.api com.ibm.dtfj.j9 com.ibm.dtfj.phd com.ibm.dtfj.sov if you have installed the IBM DTFJ feature and wish to process dumps from IBM virtual machines
  • Select com.ibm.java.doc.tools.dtfj for help for IBM DTFJ
  • Eclipse >= Neon: Select org.eclipse.equinox.ds and org.eclipse.equinox.event
  • Eclipse >= Oxygen: Select org.eclipse.equinox.event

Create a Stand-Alone RCP

See MemoryAnalyzer/Building MAT With Tycho if you want to produce a standalone MAT.

JUnit Tests

The unit tests a placed in the org.eclipse.mat.tests project. Execute the tests by right-clicking on the project and choose Run As... -> JUnit Plug-in Test.

The following VM arguments are required in the run configuration for the JUnit Plug-in Test: -Xmx850m -ea

For the org.eclipse.mat.rcp.tests project install SWTBot - API from [1].

Build Help with DITA

• Download DITA-OT 3.7 and unzip it into somewhere on your disk, e.g. C:\dita-ot-3.7. Please stick to this DITA version, it is the one with which the help pages are currently built. Using a different version results in many unnecessary file changes to the generated files (which are also committed in the git repository). [Previously DITA 1.7.4 was used.]

• In plugin org.eclipse.mat.ui.help select DitaBuild.xml and configure the runtime configuration:
  • right click Run As > Ant Build...
  • Refresh > Refresh resources upon completion. > The project containing the selected resource
  • configure the DITA directory and libraries:
    • add property dita.dir (this overrides the version in DitaBuild.xml)
      • Properties
      • Add Property
      • Variables
      • Edit Variables
      • New
        • Name: dita.dir
        • Value: the location of DITA, e.g. C:\dita-ot-3.7
        • OK
  • Alternatively to run DITA-OT from the command line
    • Set the dita directory variable, e.g. set DITA_DIR=C:\dita-ot-3.7
    • Add DITA to the path, e.g. set PATH=%DITA_DIR%\bin;%PATH%
    • change to the org.eclipse.mat.ui.help directory and run one of the following:
      • ant -f DitaBuild.xml [attempts to not change HTML files which have no content changes]
      • ant -f DitaBuild.xml -Djustnew=true [attempts to not change HTML files which have no content changes]
      • ant -f DitaBuild.xml -Djustnew=false [HTML files are as they come from DITA build, some HTML files may be changed which have no content changes]
• To modify Help documentation modify xml files
  • XML Buddy - might not be available anymore
    • Download XMLBuddy and copy a product directory (e.g., com.objfac.xmleditor_2.0_72) to the plugins directory of your Eclipse installation.
    • Configure XMLBuddy editor as described here
  • or use the XML editor from Eclipse Web Tools
    • Window > Preferences > XML > XML files > Validation > Enable markup validation
    • Window > Preferences > Validator > XML Validator > Settings > Include Group > Add Rule > File extensions : dita
    • Window > Preferences > XML > XML Catalog > User supplied entries > Add XML Catalog Element > Delegate Catalog
      • Key type to match: URI
      • Matching start string: -//OASIS//DTD DITA
      • Delegate to this XML catalog file: %DITA_DIR%/plugins/org.oasis-open.dita.v1_3/catalog.xml
      • [substitute %DITA_DIR% with the actual path]
    • Note that the validation does not seem to work with Eclipse 2022-03 any more - some previous versions did work.
  • or or use the XML editor from Eclipse Web Tools and Vex
    • It may be easier to still use the XML Editor, as the Vex editor deliberately doesn't show tags, but Vex provides DTD files for DITA, making it possible for XML validation and content assist for DITA files
• Run ant on DitaBuild.xml to build html files.

Build OQL Parser using JavaCC

• Download JavaCC 5.0 tar.gz or JavaCC 5.0 zip and unpack it.
• Copy javacc.jar to the root of the org.eclipse.mat.parser project
• In plugin org.eclipse.mat.parser select build_javacc.xml
  • right click Run As > Ant Build...
• Select package org.eclipse.mat.parser.internal.oql.parser
  • Source > Organize Imports
  • Source > Format
  • Ignore the choice conflict message and non-ASCII character message
  • Synchronize with the source repository to add the copyright header etc. back in

Creating and editing icons

Consider using EcliPaint.

For Mac, consider using icnsutil from PyPI to help build the icns file.

#!/bin/sh
cp memory_analyzer_16.png icon_16x16.png
cp memory_analyzer_32.png icon_32x32.png
cp memory_analyzer_48.png icon_48x48.png
cp memory_analyzer_64.png icon_64x64.png
cp memory_analyzer_128.png icon_128x128.png
cp memory_analyzer_256.png icon_256x256.png
icnsutil convert icon_16x16.argb icon_16x16.png
icnsutil convert icon_32x32.argb icon_32x32.png
cp icon_32x32.png icon_16x16@2x.png
cp icon_48x48.png icon_24x24@2x.png
cp icon_64x64.png icon_32x32@2x.png
cp icon_128x128.png icon_64x64@2x.png
cp icon_256x256.png icon_128x128@2x.png
icnsutil c memory_analyzer.icns icon_16x16.argb icon_16x16@2x.png icon_24x24@2x.png icon_32x32.argb icon_32x32@2x.png icon_48x48.png icon_128x128.png icon_128x128@2x.png icon_256x256.png --toc
icnsutil i memory_analyzer.icns
rm icon_*

Also see how the icons look in high-contrast mode. See Bug 342543 Icon decorators not visible in high contrast mode Also consider dark theme: Window > Preferences > General > Appearance > Theme

Building MAT with Maven/Tycho

The following page describes how Memory Analyzer (p2 repository and standalone RCP applications) can be build using Maven/Tycho: MemoryAnalyzer/Building MAT With Tycho

Testing using Docker

It is possible to run Memory Analyzer in a Docker container, which might allow testing on different Linux distributions

Testing using Windows Subsystem for Linux

It is possible to run Memory Analyzer under WSL, which might allow testing of a Linux distributions when running Windows.

Ideas for Contributions

This is just a short list of ideas. If you are missing a feature and have some time to contribute, please do not hesitate to contact us.

• Extensions to the tool adding application knowledge: MAT provides some extension points which can help you to plug-in pieces of information that give meaning to your specific object structures. It would be nice to have in MAT application knowlegde about prominent open source projects, e.g. different Eclipse components, servers like Tomcat and Glassfish, etc... Some examples:
  • using an org.eclipse.mat.api.nameResolver extension one can specify what description should be shown next to an object, similar to a toString() method. See for example bug 273915
  • using an org.eclipse.mat.api.requestResolver extension one can add to the leak report information about what a threads was executing, e.g. tell which HTTP request it was processing, list the URL, parameters, etc... For example bug 318989 proposes an extension which points to the ruby script a thread is executing
  • using an org.eclipse.mat.api.query extension one can add a useful query which will be available all the other queries/commands seen in the tool. See for example bug 256154
• Documentation
• Unit Tests

Contributing code

Eclipse Memory Analyzer uses Gerrit for contributions, including from committers. Contributions are pushed to Gerrit using these URLs, not directly to Git. After an approval process (including the Eclipse IP process) by MAT committers, the contributions are pushed by Gerrit to the main Eclipse Git repository. The code is then build by Jenkins. See also Development_Resources/Contributing_via_Git.

Example Workflow

• Open a bug at https://bugs.eclipse.org/bugs/enter_bug.cgi?product=MAT
• Open Eclipse's Git perspective
• Right click on the repository and click Pull to get the latest changes
• Right click on the repository and click Switch To > New Branch and enter an arbitrary branch name (perhaps with the bug number) and click Finish
• Make your code changes
• Format your code to the project standards using MAT code style format
• Run org.eclipse.mat.tests as a Junit Plugin Test and org.eclipse.mat.rcp.tests as an SWTBot test and ensure all pass
• Switch to the Team Synchronizing perspective and review your changes
• Add the changes you want to commit to the index (e.g. right click on the project and click Add to Index)
• Right click on the projects/files and click Commit...
• Use the following commit message template:

Bug $BUGID Commit message...

Example:

Bug 497127 Handle copying heapdumps greater than 2GB on certain platforms.

• Click Commit (do not click Commit and Push)
• In the Git perspective, expand org.eclipse.mat > Remotes, then right click on origin, then click Gerrit Configuration..., set destination branch to master and click Finish.
• Right click on org.eclipse.mat and click Push to Gerrit... and click Finish
• The results dialog should show the Gerrit link at the bottom such as https://git.eclipse.org/r/76397
• In the Git perspective, right click the repository and click Switch To > master (now you're back on master without your changes)
• Gerrit requires two code review votes and one verify vote. For trivial changes, you may give the +2/+1 yourself. Otherwise, ideally, two other committers would review and approve your change, but given the size of the team and their limited availability, you may also give your own +1. In the trivial case, add a comment such as "Simple patch; judging it doesn't require others' review so adding +2 code review". To request that a committer reviews your change, click Add next to Reviewers and find the name
• Once the +2 Code Review/+1 Verified exists, a "Submit" button will show up which you can click to push the change to master.
• The bug report will be updated automatically
• Change the bug report to Resolved after testing the changes from a nightly build

If you are asked to modify your Gerrit change, you will need to amend your commit and add a patch set: https://gerrit-review.googlesource.com/Documentation/intro-user.html#upload-patch-set

Note in particular that the amended commit should have "\nChange-Id: ..." at the bottom which matches the Gerrit change.

If you have the Gerrit commit hook installed, the steps to submit a new patch set are:

• Make your changes in the original patch set branch
• git add your changed files
• git commit --amend
• git push origin HEAD:refs/for/master

Uses Mylyn for handling Bugzilla tasks

From the Task Repositories view add your Bugzilla user ID and password:

• Server: https://bugs.eclipse.org/bugs
• Label: Eclipse.org
• User ID: my_name@example.com
• Password: ********

From the Task List then create a Query: New > Query > Eclipse.org > Create query using form > Next

• Product: MAT
• Status: UNCONFIRMED NEW ASSIGNED REOPENED

This lets you handle Bugzilla from inside Eclipse.

Writing plugins for the Memory Analyzer

If you want to write a plugin for the Memory Analyzer, you can find information on the following pages:

• MemoryAnalyzer/Reading Data from Heap Dumps
• MemoryAnalyzer/Extending Memory Analyzer
• MemoryAnalyzer/Adding a new heap dump format to the Memory Analyzer

You should develop and test your plug-ins using the Eclipse environment, making sure that your plug-in is listed in the run configuration. Memory Analyzer uses the p2 installer Equinox p2 Getting Started. To get your plug-in installed in standalone version of MAT then you need to build an Eclipse feature or update site including your plug-in, then install the feature or update site into MAT.

Help > Install New Software > Add > Archive > your exported feature

You may need to deselect 'Group items by category' if your feature does not have categories.

If you wish to import an plug-in without building a feature then you need to create a dropins folder under the mat directory in MAT, put your plug-in jar there, then restart MAT with

MemoryAnalyzer -console 

and type

start org.eclipse.equinox.p2.reconciler.dropins 

in the console so that the p2 installer will look in the plugins directory for your plug-ins.

New version development process

• Document new version at MAT project page
• Create Bugzilla entries for target milestone Bugzilla Manager
• Update references to old release in the code e.g. 1.X -> 1.Y excluding update sites
  • See pom.xml e.g. <version>1.9.1-SNAPSHOT</version>
  • See manifest.mf e.g. <Bundle-Version: 1.9.1.qualifier
  • See feature.xml, excluding updateSiteName
  • See org.eclipse.mat.ui.rcp about.mappings
  • org.eclipse.mat.product mat.product
  • org.eclipse.mat.ui.rcp.feature rootfiles/.eclipseproduct (hidden file, may need to use navigator view)

• Develop features and fix bugs
• If a plugin depends on new function in another plugin, update the dependency version in manifest.mf
• If creating a new plugin, add it to the JavaDoc build process in extrabuild.xml, use package-info.java to mark packages as not API or API as appropriate. Consider carefully adding new APIs.
• If the Java version changes then the minor version must increase, also change:
  • .classpath
  • .settings/org.eclipse.jdt.core.prefs
  • manifest.mf Bundle-RequiredExecutionEnvironment: J2SE-1.5
  • Update org.eclipse.ui.help extrabuild.xml for the new JavaDoc compile level, and for the link to the Java class library documentation
  • Consider keeping org.eclipse.mat.ibmdumps at a lower level as it uses classes for the exec dump provider and the attach jar which may be executed on lower level JVMs.
• Update copyright date in source code if updated in a new year
• If the RCP is to be built against a newer version of Eclipse, then:
  • create a new target platform in org.eclipse.mat.targetdef
  • update org.eclipse.mat.ui.help extrabuild.xml to add a link to the Eclipse help for the platform
  • create a new /org.eclipse.mat.product/mat-<Eclipse-rel>.product file - normally use the same basename as for the target
  • create a new /org.eclipse.mat.product/mat-<Eclipse-rel>.p2.inf file - normally use the same basename as for the target
• Check for regressions/changes in report outputs using the regression test suite
  • Check out the previous release from Git
  • Get it compiled - may need to change target platform
  • In org.eclipse.mat.test, run the org.eclipse.mat.tests.application with

-regression ./dumps "-Xmx500m -DMAT_HPROF_DUMP_NR=#1 --add-exports=java.base/jdk.internal.org.objectweb.asm=ALL-UNNAMED" . This will run the tests, and establish a baseline if one does not exist

• • Switch back to master
  • Rerun the test which will detect any changes. Examine report.xml to understand whether the changes are expected.
• Towards the end, change the update site references
• Update copyright date in feature.properties etc. if feature/plugin updated in a new year
  • See feature.xml, including updateSiteName
• Also write a New and Noteworthy document replacing the previous release, and add a link to the old document on the MAT website. This should be done in org.eclipse.mat.help in noteworthy.dita, and take the generated noteworthy.html, modify it if needed and add to the website.
• Follow MemoryAnalyzer/Contributor Reference#Simultaneous release policies
• After release create Bugzilla entry for the new version Bugzilla Manager
• Add a new Babel definition so the messages files could be translated.

Simultaneous release policies

• Create a release record in Eclipse Memory Analyzer project page
• For reference, read contribute to the Simultaneous Release Build
• Build to copy update site build to SimRel location.
• Follow the SimRel process to update mat.aggrcon in the SimRel build.
• Preserve the tycho-build-nightly as 'Keep forever' and label it with the release build e.g. 'Photon RC2'
• The MAT configuration file which is updated using Git in GitHub.com/eclipse-simrel to match the SimRel location.
• Tag in Git with 'R_1.X.Y' the source used to generate the final build for a release. See MAT Git Refs
• Complete the Eclipse release process, including getting a review if needed at Eclipse Memory Analyzer project page
• To release, also run the Stand-alone packaging build, notarize the Mac x86_64 and aarch64 .dmg files and copy the results to the download site using promote release.
• Add the version name '1.XX' to Bugzilla manager so users can report bugs against the new version
• Also release on Eclipse Marketplace
• Also consider archiving some old releases

MAT Policies - to satisfy SimRel/Simultaneous_Release_Requirements/Appendix

• Ramp Down Plan
• Retention Policy
• API Policy
• Capabilities

Download Statistics

Eclipse committers once logged in at accounts.eclipse.org can see download statistics at Download Stats. These are from downloads via the Find a Mirror script for stand-alone MAT and from p2 downloads from an update site. Search for '/mat/' for mirror downloads and 'org.eclipse.mat.api' for p2 downloads.

Maintaining MAT Website

See MemoryAnalyzer/Contributor Reference/Website