EclipseLink/Development/JPA 2.0/ordered lists

Persisting Order

JPA 2.0 Root | bug 249037

Issue Summary

JPA 2.0 specification has added functionality requiring the provider to persist list item indexes when mapped by the user. The provider must also track changes to an index.

See JPA 2.0 ED section 10.1.36 for details.

Additional requirements

• Solution should enable the storage of duplicates in ManyToMany lists. See bug 256978.
• Must offer flexibility in the naming and type of the database index columns to ensure users implementing their own work-arounds can easily upgrade to this implementation

General Solution

Ordered List support is currently available in EclipseLink. Order of collection items is tracked and updates and merges occur as a result of changes. What this feature requires is that an additional field be managed by EclipseLink that is not mapped in the object model, similar to the Uni-directional OneToMany mappings. This additional field will store the index of each item in the collection.

As all collection mappings will need to support this functionality the best approach is to make CollectionMapping able to write indexes. ChangeRecords are already aware of order changes but the mappings are unable to write the changes.

It would be favourable to be able to restrict the number of updates of the target row to one even if the target object has other changes. This may be possible by allowing the mapping to force the write of the target object and add index to the query.

There is a potential enhancement that allows the provider to fabricate artificial indexes to allow for efficient insertion. For example if there were 3 items in the list EclipseLink could assign index 1, 3, and 9 to these objects. Order would be maintained and should another instance be inserted into the list only one row update would be required instead of the minimal two. This should only be considered as a future enhancement to this feature.

Requirements

Main use case

The Collection used in ToMany relationship should be a List.

public class Employee {
    private List<Dealer> dealers;
...
    @OneToMany(cascade={PERSIST, MERGE})
    @JoinColumn(name="FK_EMP_ID")
    @OrderColumn(name="ORDER")
    public List<Dealer> getDealers() {
        return dealers;
    }
...
}

List members saved together with their indexes in the List:

Employee employee = new Employee();
Dealer dealerA = new Dealer("A");
employee.getDealers().add(dealerA);
Dealer dealerB = new Dealer("B");
employee.getDealers().add(dealerB);
Dealer dealerC = new Dealer("C");
employee.getDealers().add(dealerC);
em.persist(employee);

DEALER_ID   FK_EMP_ID    NAME     ORDER
33          5            A        0
34          5            B        1
35          5            C        2

The indexes updated if the List member(s) removed:

employee.getDealers().remove(dealerB);

DEALER_ID   FK_EMP_ID    NAME     ORDER
33          5            A        0
35          5            C        1

the order changed:

employee.getDealers().remove(dealerA);
employee.getDealers().add(dealerA);

DEALER_ID   FK_EMP_ID    NAME     ORDER
33          5            A        1
35          5            C        0

new member(s) added:

Dealer dealerD = new Dealer("D");
employee.getDealers().add(dealerD);

DEALER_ID   FK_EMP_ID    NAME     ORDER
33          5            A        1
35          5            C        0
35          5            D        2

Open Issues

Non Contiguous Index values

What if the collection is altered directly (not through Eclipselink) in the db so that the indexes are no longer a contiguous sequence of numbers from 0 to size-1?

Some elements removed from the beginning or the middle:

DEALER_ID   FK_EMP_ID    NAME     ORDER
35          5            C        0
35          5            D        2

Some elements' order is null;

DEALER_ID   FK_EMP_ID    NAME     ORDER
35          5            C        
35          5            D

Some elements have equal ordered id values

DEALER_ID   FK_EMP_ID    NAME     ORDER
35          5            C        3
35          5            D        3

How do we read back such "damaged' lists into Java app.?

1. Should we throw an exception?
2. Try to cure the list by assigning (sometimes randomly) the indexes?
3. Insert null element into the List that corresponds to missing index (note that then the list size will be greater than the real number of objects)?

One possible approach is to verify the previous order value each time the order value is changed, for instance, in case the order value should be changed drom to to three, instead of simply assigning the new value:

  UPDATE DEALER SET ORDER = 3 WHERE DEALER_ID = 35

verify that the old value was correct:

  UPDATE DEALER SET ORDER = 3 WHERE DEALER_ID = 35 AND ORDER = 2

In case either no rows or more than one rows were updated - update all the rows for all Dealers referencing this Employee.

The approach is not perfect: it doesn't do anything when the last object is added or removed. Also it's incompatible with OptimisticLocking on the target.

Indexing new elements in un-instantiated IndirectLists

When an IndirectList has new items added it does not force the entire list to be loaded. This means that during commit you need to assign new index values without necessarily having all of the contents of the list loaded. In my EclipseLink/Examples/JPA/Collectionordering example for EclipseLink 1.1 i am addressing this with a combination of a query for the max index value as well as forcing an optimistic locking update of the parent object if it supports it.

OrderColumn table

OrderColumn annotation defines a table in which the order column is located. When the table is not specified, by default the table used is either join table (if exists) or otherwise the first table of the target entity. Which tables explicitly specified in OrderColumn annotation we should support?

1. The secondary table of an Entity (i.g. SALARY table in Employee example) - should be supported.
2. In case the join table exists - should we support target table being specified (i.g. ManyToMany Employee -> Project, should support ORDER field in PROJECT table - though it's EMP_PROJ join table by default)?
   1. Note that this may cause essential variation in behaviour, for instance duplicated could be only supported if the order field is in join table).
3. User specified a table which is neither join table nor part of this entity (nor CollectionTable in ElementCollection). We should NOT support this, ask the spec. to explicitly prohibit this case.

Duplicate support

1. Should duplicate support be part of this feature?
2. Should it be done only for ManyToMany or for all cases when it's possible (all cases with join table)?
3. Note that duplicate support complicates privately owned case - until now the removal of privately owned child meant that it should be deleted, now we'll have to verify whether there's a suplicate of the removed child still left in the collection - in that case the removed child should NOT be deleted.

"Two-way" order support

Should we (could we?) support two OrderFields used by two ManyTopMany mappings pointing at each other (Employee.projects and Projects.employees; say add ORDER_EMP and ORDER_PROJ fields to the join table)?

Target optimistic locking

If the target of OrderColumn-annotated list uses OptimisticLocking - should we change version of the target object each time the order value is changed?

UnidirectionalOneToManyMapping is a similar case - by default we change the target version each time it changes its collection; also there are two boolean flags defined in the mapping which allow to alter that behaviour: shouldIncrementTargetLockValueOnAddOrRemoveTarget and shouldIncrementTargetLockValueOnDeleteSource.

Work Required

1. Develop model for testing

   approx 5 days - including all collection mapping types and maps

2. Update EclipseLink

   approx 10 days - prototype mappings writing additional fields

   approx 10 days - implementing mapping support

   approx 3 days - JPQL INSERT() updates

   approx 3 days - verifying change detection/tracking support

   approx 2 days - Process annotations/XML