Skip to main content
Jump to: navigation, search

MemoryAnalyzer/OQL

< MemoryAnalyzer
Revision as of 17:16, 20 November 2019 by Andrew johnson.uk.ibm.com (Talk | contribs) (OQL (Memory Analyzer) versus SQL (MAT/Calcite))

Object Query Language

Object Query Language is an SQL like language used by Memory Analyzer for exploring a heap dump. There is documentation in the help but this wiki allows newer features to be explained and discussed before the documentation is updated.

Simple

SELECT * FROM java.lang.String

Displays all String objects as a tree.

SELECT s as String,s.value as "characters" FROM java.lang.String s

Displays all String objects as a table.

SELECT s as String,s.value as "characters", inbounds(s),inbounds(s).@length FROM java.lang.String s
String                          |characters                                 |inbounds(s)| inbounds(s).@length
--------------------------------------------------------------------------------------------------------------
java.lang.String [id=0x22e58820]|char[] [id=0x22e60f50;length=16;size=48]   |[I@620f7a39|                   1
java.lang.String [id=0x22e59150]|char[] [id=0x22e62ff0;length=6;size=24]    |[I@1f7b8d59|                   1
java.lang.String [id=0x22e5b560]|char[] [id=0x22e6b730;length=537;size=1088]|[I@28551755|                   1
--------------------------------------------------------------------------------------------------------------

There are two sorts of objects encountered with OQL, IObject which represent Java objects in the snapshot and regular Java objects generated by OQL processing.

java.lang.String [id=0x22e58820] is a IInstance representing a String from the snapshot.

char[] [id=0x22e60f50;length=16;size=48] is an IPrimitiveArray representing a character array from the snapshot.

[I@620f7a39 is a regular Java integer array holding a several of ints which are the object IDs Memory Analyzer uses to represent IObjects in the snapshot.

OQL (Memory Analyzer) versus SQL (MAT/Calcite)

As well as the built-in OQL, there is an extension plug-in for MAT called MAT Calcite which adds SQL processing

Topic OQL SQL
General syntax
SELECT s FROM java.lang.String s
SELECT s.this FROM "java.lang.String" s
Built-in functions
SELECT toString(s), classof(s), 
s.@objectAddress, s.@usedHeapSize, s.@retainedHeapSize
FROM java.lang.String s
SELECT toString(s.this),getType(s.this),
getAddress(s.this),shallowSize(s.this),retainedSize(s.this) 
FROM "java.lang.String" s
More functions
SELECT h, h[0:-1].size(), h.table, 
h.table.@length, h.modCount, h.getField("modCount") 
FROM java.util.HashMap h
SELECT h.this,getSize(h.this),h.this['table'],
LENGTH(h.this['table']), h.this['modCount'], getField(h.this,'modCount') 
FROM "java.util.HashMap" h
Comments /* multi-line comment */ /* multi-line comment */ Single line comment // comment

OQL/Calcite provides advanced SQL functions such as JOIN (INNER JOIN,LEFT JOIN,RIGHT JOIN,FULL JOIN), CROSS JOIN, GROUP BY, ORDER BY. Some can be simulated by OQL with a bit of work.

Enhancements in November 2019

Various updates and enhancements have been made to OQL under 552879: OQL enhancements for sub-selects, maps, context providers, DISTINCT. These are available from snapshot builds for testing and are subject to change. Please comment on the bug, forum or development mailing list

Some of the aims of these changes were to permit more complex queries such as:


SELECT DISTINCT

`DISTINCT` used to just operate on the results of a query if it returned objects rather than general select items.

SELECT DISTINCT OBJECTS classof(s) FROM "java.lang..S*" s
SELECT * FROM OBJECTS "java.lang.S.*"

`DISTINCT` now also operates on SELECTs with select items. It uses the whole row considered as a list as the item to be considered as distinct. It also uses the optimization that the input FROM items are also considered as being distinct (either as ints/IObjects or more general `FROM` items).

SELECT DISTINCT classof(s) FROM "java.lang..S*" s

Sub SELECT with select items

sub-selects were permitted where the sub-select returned an object list.

SELECT v, v.@length FROM OBJECTS ( SELECT OBJECTS s.value FROM java.lang.String s  ) v

sub-selects are now permitted which have select items.

SELECT v,v.s,v.val FROM OBJECTS ( SELECT s,s.value as val FROM java.lang.String s  ) v
Row                                                                                |Object                          |Array
--------------------------------------------------------------------------------------------------------------------------------------------------------------
{s=java.lang.String [id=0x26ba8a30], val=char[] [id=0x26ba8a48;length=14;size=40]} |java.lang.String [id=0x26ba8a30]|char[] [id=0x26ba8a48;length=14;size=40]
{s=java.lang.String [id=0x26ba8998], val=char[] [id=0x26ba89b0;length=56;size=128]}|java.lang.String [id=0x26ba8998]|char[] [id=0x26ba89b0;length=56;size=128]
{s=java.lang.String [id=0x26ba8160], val=char[] [id=0x26ba8178;length=56;size=128]}|java.lang.String [id=0x26ba8160]|char[] [id=0x26ba8178;length=56;size=128]
{s=java.lang.String [id=0x26b9d390], val=char[] [id=0x26b9d3a8;length=15;size=48]} |java.lang.String [id=0x26b9d390]|char[] [id=0x26b9d3a8;length=15;size=48]
{s=java.lang.String [id=0x26b9d358], val=char[] [id=0x26b9d370;length=8;size=32]}  |java.lang.String [id=0x26b9d358]|char[] [id=0x26b9d370;length=8;size=32]
--------------------------------------------------------------------------------------------------------------------------------------------------------------

The outer select processes the result of the sub-select row by row, with a single RowMap `Map` object representing the row. The key/value pairs are the sub-select items with the sub-select column names as the keys. If the keys are standard identifiers, i.e. generally alpha-numeric then attribute processing can be used as `v.s` rather than having to to `v.get("s2")` which can still be used, perhaps for column names with spaces.

The whole sub-select continues to return a `CustomTableResultSet` which is an `IResultTable` but this has been enhanced to also be a `List` of `RowMap` items. It is quite hard to operate in OQL on the whole result as if it is supplied to an outer select then the `Iterable` nature means it will be processed row by row.

SELECT * FROM OBJECTS ( SELECT s, s.value AS val FROM java.lang.String s ) v
[{s=java.lang.String [id=0x26ba8a30], val=char[] [id=0x26ba8a48;length=14;size=40]}, {s=java.lang.String [id=0x26ba8998], val=char[] [id=0x26ba89b0;length=56;size=128]}, {s=java.lang.String [id=0x26ba8160], val=char[] [id=0x26ba8178;length=56;size=128]}, {s=java.lang.String [id=0x26b9d390], val=char[] [id=0x26b9d3a8;length=15;size=48]}, {s=java.lang.String [id=0x26b9d358], val=char[] [id=0x26b9d370;length=8;size=32]}, {s=java.lang.String [id=0x26b9d318], val=char[] [id=0x26b9d330;length=11;size=40]}, {s=java.lang.String [id=0x26b9d2e8], val=char[] [id=0x26b9d300;length=4;size=24]}, {s=java.lang.String [id=0x26b9c758], val=char[] [id=0x26b9c770;length=21;size=56]}, {s=java.lang.String [id=0x26b9c6c8], val=char[] [id=0x26b9c6e0;length=13;size=40]}, {s=java.lang.String [id=0x26b9c690], val=char[] [id=0x26b9c6a8;length=8;size=32]}, ...

Shows the whole table as a list

SELECT eval((SELECT * FROM OBJECTS ( SELECT s, s.value AS val FROM java.lang.String s ) v))[3] FROM OBJECTS 0
eval((SELECT * FROM OBJECTS ( SELECT s, s.value AS val FROM java.lang.String s  ) v ))[3]
-----------------------------------------------------------------------------------------
{s=java.lang.String [id=0x26b9d390], val=char[] [id=0x26b9d3a8;length=15;size=48]}
-----------------------------------------------------------------------------------------

Processes the whole table as a select item.

Context Menu for object columns

If a column appears to hold heap objects, or lists or arrays of heap objects, then the context menu now offers a choice to process that column's item of the selected rows.

SELECT s AS String, s.value AS "Char array", inbounds(s) AS Inbounds FROM java.lang.String s
String                          |Char array                               |Inbounds
--------------------------------------------------------------------------------------
java.lang.String [id=0x26ba8a30]|char[] [id=0x26ba8a48;length=14;size=40] |[I@6ad112de
java.lang.String [id=0x26ba8998]|char[] [id=0x26ba89b0;length=56;size=128]|[I@18a0721b
java.lang.String [id=0x26ba8160]|char[] [id=0x26ba8178;length=56;size=128]|[I@2ae2fa13
--------------------------------------------------------------------------------------

Context Menu:

SELECT ... s
The entire row - based on the underlying object s. `SELECT s AS String, s.value AS "Char array", inbounds(s) AS Inbounds FROM OBJECTS 20798,20796,20793 s`
String
Just the String item in column 'String'. `SELECT s AS String FROM OBJECTS 20798,20796,20793 s`
Char array
Just the char array. `SELECT s.value AS "Char array" FROM OBJECTS 20798,20796,20793 s`
Inbounds
All the inbounds as heap objects. `SELECT inbounds(s) AS Inbounds FROM OBJECTS 20798,20796,20793 s`


The context menu has a `Copy > OQL Query` option which returns an OQL query representing the selected rows and appropriate column.

The report plug-in which converts result tables to HTML now uses the context menu name to match with the table column to put HTML linksin the correct place across the columns in the table rather than always in the first column. This also applies for other queries, so the system properties query when used in a report has in-place links for keys and values.

Map processing

Map heap objects in the heap dump can be now accessed using array notation, returning Map.Entry items. Previously array access returned any Map.Entry heap objects in the heap dump for the map, which could then be used to find the key and value via the `key` and `value` fields. Not all maps have entry objects, so the new system means that `getKey()` and `getValue()` can be used to access the keys and values.

Question: when array access returns map entry objects, should those objects have a fake object ID of the actual map heap object, or of the Map.Entry heap object if one was available? This affects all collection extraction of maps, not just OQL. See [1]

SELECT h AS map, (SELECT e.getKey() AS key, e.getValue() AS value FROM OBJECTS ${h}[0:-1] e ) AS kv FROM java.util.HashMap h WHERE (h[0:-1].size() > 0)
map                              |kv
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
java.util.HashMap [id=0x22f49970]|[{key=java.lang.String [id=0x22f44658], value=java.lang.String [id=0x22f44670]}, {key=java.lang.String [id=0x22f44688], value=java.lang.String [id=0x22f446a0]}]
java.util.HashMap [id=0x22f49948]|[{key=java.lang.String [id=0x22f44628], value=java.lang.String [id=0x22f44640]}]
java.util.HashMap [id=0x22f49920]|[{key=java.lang.String [id=0x22f445c8], value=java.lang.String [id=0x22f445e0]}, {key=java.lang.String [id=0x22f445f8], value=java.lang.String [id=0x22f44610]}]
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Extracts the map and a list of key / values pairs.

Question: Should MAT use the select clause or the AS column name as the key for the RowMap and subsequent access, or should it just use a simple name if given as part of the select, and a generated term such as `Expr1000` or `EXPR$0` (Apache Calcite) if more complex? Is an autogenerated term required in the case of duplicated select items or column names?

Flattening

The second column is a list of key/value pairs. It would be nice to process these further, and auto-flattening is one way to achieve this. If a sub-select returns a RowMap containing values which are lists or arrays, then auto-flattening splits that RowMap into multiple RowMaps, one for each entry of the list or array. Other objects are just repeated in the RowMap. If there are multiple lists or arrays of different lengths then items beyond the end of the list or array are replaced by null.

Question: This is very experimental and is intended to be a basic alternative to SQL JOIN. Is auto-flattening of sub-selects the way to go, or should it operated on FROM method calls as well or an alternative. Should there be a `flatten((SELECT ...))` function, or a `flatten()` method on the result table?

The aim is to achieve a result such as

Map                              |Key                             |Value
----------------------------------------------------------------------------------------------------------------------------------------------------------------
java.util.HashMap [id=0xe35e96a8]|java.lang.String [id=0xe35ef478]|com.sun.management.internal.PlatformMBeanProviderImpl$4 [id=0xe35cbf68]
java.util.HashMap [id=0xe35e96a8]|java.lang.String [id=0xe35e96d8]|jdk.management.jfr.internal.FlightRecorderMXBeanProvider$SingleMBeanComponent [id=0xe35e98b0]
java.util.HashMap [id=0xe35ce190]|java.lang.String [id=0xe35c1950]|java.lang.Object [id=0xe0c145f0]
java.util.HashMap [id=0xe35ce190]|java.lang.String [id=0xe35c1900]|java.lang.Object [id=0xe0c145f0]
java.util.HashMap [id=0xe35cbde0]|java.lang.String [id=0xe35cbc70]|java.lang.Object [id=0xe0c145f0]
----------------------------------------------------------------------------------------------------------------------------------------------------------------


  1. SELECT z.map as Map, z.kv.key as Key, z.kv.value as Value FROM OBJECTS ( SELECT h AS map, (SELECT e.getKey() AS key, e.getValue() AS value FROM OBJECTS ${h}[0:-1] e ) AS kv FROM java.util.HashMap h WHERE (h[0:-1].size() > 0) ) z
  2. SELECT z.map AS Map, z.kv.key AS Key, z.kv.value AS Value FROM OBJECTS (eval(( SELECT h AS map, (SELECT e.getKey() AS key, e.getValue() AS value FROM OBJECTS ${h}[0:-1] e ) AS kv FROM java.util.HashMap h WHERE (h[0:-1].size() > 0))) ) z
  3. SELECT z.map AS Map, z.kv.key AS Key, z.kv.value AS Value FROM OBJECTS (flatten((SELECT h AS map, (SELECT e.getKey() AS key, e.getValue() AS value FROM OBJECTS ${h}[0:-1] e ) AS kv FROM java.util.HashMap h WHERE (h[0:-1].size() > 0))) ) z
  4. SELECT z.map AS Map, z.kv.key AS Key, z.kv.value AS Value FROM OBJECTS (eval(( SELECT h AS map, (SELECT e.getKey() AS key, e.getValue() AS value FROM OBJECTS ${h}[0:-1] e ) AS kv FROM java.util.HashMap h WHERE (h[0:-1].size() > 0)).flatten()) ) z

Question: How should flattening handle lists or arrays with no items? Should the row be omitted, or should a row with no entries for the array be generated as a null for the item, or should the empty array be left unchanged? It is hard to add back a row later, but it is possible to filter rows with null or an empty list. It is hard to tell an empty list/array from one which contains one entry which is the same as the default value OQL chose (null or an empty list/array).


Here is another example of flattening - a query to see if any child of a parent does not have a back reference to the parent.

SELECT group AS Group, thread AS Thread FROM OBJECTS ( SELECT t AS group, t.threads[0:-1] AS thread FROM java.lang.ThreadGroup t  ) WHERE ((thread != null) and (thread.group != group))

This selects all the java.lang.ThreadGroup objects and then generates rows with two columns, the group and a list of the child threads. This is then flattened to a rows of the group and a single child thread, where the select then checks for a non-null child Thread and a child which does not point back to the ThreadGroup. Note here the omission of the alias name before the 'WHERE' as it is not necessary - 'group' and 'thread' in the outer select do not need to be qualified with an alias name.


JOIN operations

OQL does not have SQL-style JOIN operations apart from UNION. With flattening it is possible to simulate some of these operations, but the statements required are more complex.

Consider a JOIN on `java.lang.Integer` and `java.lang.Long` based on the value fields of both.

CROSS JOIN

This operation generates every combination of the left and right tables (sets of objects), so can generate a huge sized result table.

SELECT z.i AS Integer, z.i.value AS "Integer value", z.lv.l AS Long, z.lv.l.value as "Long value" 
FROM OBJECTS ( SELECT i, (SELECT l FROM java.lang.Long l ) AS lv FROM java.lang.Integer i  ) z

This selects all the `java.lang.Integer` heap objects, then for each Integer heap object then generates a row with the object and a list of all the java.lang.Long objects, then flattens the rows. Each flattened row has one Integer heap object (accessed via 'z.i' or 'i') and one Long heap object (accessed via 'z.lv.l' or 'lv.l'). The objects in one row do not necessarily match in value.

Compare to MAT Calcite (SQL):

SELECT i.this,i.this['value'] AS "Integer value", l.this,l.this['value'] AS "Long value" 
FROM "java.lang.Integer" i CROSS JOIN "java.lang.Long" l

LEFT JOIN / LEFT OUTER JOIN

This operation generates every row from the left table (set of objects) and includes in that row any corresponding row from the right table. The result table is the same size as the left.

SELECT z.i AS Integer, z.i.value AS "Integer value", z.lv.l AS Long, z.lv.l.value as "Long value"  
FROM OBJECTS ( SELECT i, (SELECT l FROM java.lang.Long l WHERE (l.value = i.value)) AS lv FROM java.lang.Integer i  ) z

This selects all the `java.lang.Integer` objects, then for each Integer then generates a row with the object and a list of all the java.lang.Long objects with the same value, then flattens the rows. Each flattened row has one Integer heap object (accessed via 'z.i' or 'i') and possibly one matching Long heap object or null (accessed via 'z.lv.l' or 'lv.l').

Compare to MAT Calcite (SQL):

SELECT i.this,i.this['value'] AS "Integer value", l.this,l.this['value'] AS "Long value" 
FROM "java.lang.Integer" i LEFT JOIN "java.lang.Long" l ON i.this['value']+0 = l.this['value']+0

INNER JOIN

This operation generates every row from the left table (set of objects) which has a matching row in the right table (set of objects). The result table is no bigger than the smaller of the left and right tables.

SELECT z.i AS Integer, z.i.value AS "Integer value", z.lv.l AS Long, z.lv.l.value as "Long value" 
FROM OBJECTS ( SELECT i, (SELECT l FROM java.lang.Long l WHERE (l.value = i.value)) AS lv FROM java.lang.Integer i  ) z 
WHERE (z.lv != null)

This selects all the `java.lang.Integer` objects, then for each Integer then generates a row with the object and a list of all the java.lang.Long objects with the same value, then flattens the rows and excludes any row without a java.lang.Long value.

SELECT z.iv.i AS Integer, z.iv.i.value AS "Integer value", z.l AS Long, z.l.value as "Long value" 
FROM OBJECTS ( SELECT (SELECT i FROM java.lang.Integer i WHERE (i.value = l.value)) AS iv, l FROM java.lang.Long l  ) z 
WHERE (z.iv != null)

This selects all the `java.lang.Long` objects, then for each Long then generates a row with the object and a list of all the java.lang.Integer objects with the same value, then flattens the rows and excludes any row without a java.lang.Integer value. Each flattened row has one Integer heap object (accessed via 'z.i' or 'i') and one Long heap object (accessed via 'z.lv.l' or 'lv.l') which matches by the 'WHERE (i.value = l.value)' clause.

Compare to MAT Calcite (SQL):

SELECT i.this,i.this['value'] AS "Integer value", l.this,l.this['value'] AS "Long value" 
FROM "java.lang.Integer" i INNER JOIN "java.lang.Long" l ON i.this['value']+0 = l.this['value']+0

RIGHT JOIN / RIGHT OUTER JOIN

This operation generates every row from the right table (set of objects) and includes in that row any corresponding row from the left table which matches. The result table is the same size as the right.

SELECT z.iv.i AS Integer, z.iv.i.value AS "Integer value", z.l AS Long, z.l.value as "Long value" 
FROM OBJECTS ( SELECT (SELECT i FROM java.lang.Integer i WHERE (i.value = l.value)) AS iv, l FROM java.lang.Long l  ) z

This selects all the `java.lang.Long` objects, then for each Long then generates a row with the object and a list of all the java.lang.Integer objects with the same value, then flattens the rows. Each flattened row has one Long heap object (accessed via 'z.l' or 'l') and possibly one matching Integer heap object or null (accessed via 'z.iv.i' or 'iv.i').

Compare to MAT Calcite (SQL):

SELECT i.this,i.this['value'] AS "Integer value", l.this,l.this['value'] AS "Long value" 
FROM "java.lang.Integer" i RIGHT JOIN "java.lang.Long" l ON i.this['value']+0 = l.this['value']+0

FULL OUTER JOIN

This operation generates has a row for every row from the left table (set of objects) and every row from the right table (set of objects), but when the left table has a row with the same value as a row from right table they will be included in the same output row. The result table is at least as big as the bigger of the left table and right table.

SELECT z.i AS Integer, z.i.value AS "Integer value", z.lv.l AS Long, z.lv.l.value as "Long value" 
FROM OBJECTS ( SELECT i, (SELECT l FROM java.lang.Long l WHERE (l.value = i.value)) AS lv FROM java.lang.Integer i  ) z 
UNION (
SELECT z.iv.i AS Integer, z.iv.i.value AS "Integer value", z.l AS Long, z.l.value as "Long value" 
FROM OBJECTS ( SELECT (SELECT i FROM java.lang.Integer i WHERE (i.value = l.value)) AS iv, l FROM java.lang.Long l  ) z WHERE (z.iv = null)
)

This does a LEFT JOIN / LEFT OUTER JOIN then combines the rows with a list of all java.lang.Long objects which do not have a corresponding java.lang.Integer object. Each row contains either an Integer heap object or a Long heap object or both.

Compare to MAT Calcite (SQL):

SELECT i.this,i.this['value'] AS "Integer value", l.this,l.this['value'] AS "Long value" 
FROM "java.lang.Integer" i FULL OUTER JOIN "java.lang.Long" l ON i.this['value']+0 = l.this['value']+0

Bug fixes

  • When there is a union queries with a select which returns no items, that select was then omitted from the command window.
SELECT s FROM java.lang.String s UNION (SELECT s FROM java.lang.Missing s)
this was then redisplayed as
SELECT s FROM java.lang.String s
  • Context dependency fix. OQL processing optimizes some queries by detecting that some parts are not context dependent and will evaluate the same each time, so can just be evaluated once. This processing was not correct for sub-select. Now a sub-select in a select item will be correctly re-evaluated if required.
  • Progress monitoring has been improved so the progress monitor bar graph better shows how much more work needs to be done to complete a query. Also, cancelling a long running OQL query works more swiftly.

Back to the top