Latest revision as of 10:00, 25 March 2010

Design Specification: MOXy Support for Dynamic Persistence

ER 296967

Document History

Project Overview

The goal of this feature is to enable users of EclipseLink JAXB to perform typical JAXB operations without having real .class files available for their domain objects. Users will pass in some form of metadata (XML Schema, EclipseLink Deployment XML or External Metadata) to create a DynamicJAXBContext, and this metadata will be used to construct DynamicEntity objects representing the domain classes.

The ultimate purpose of this feature is to support an end-to-end JPA<->JAXB solution. EclipseLink JPA already has some support for dynamic persistence and this work will provide a path from dynamic JPA to JAXB.

Some initial groundwork was included in EclipseLink 2.0, more information is available here.

APIs for bootstrapping from EclipseLink Project and XML Schema will be included in EclipseLink 2.1.0, with support for bootstrapping from EclipseLink OXM to follow in a future release.

Concepts

Dynamic Persistence

The core of this feature is the dynamic persistence support that was initially added for JPA (found in org.eclipse.persistence.dynamic). To obtain a "dynamic" project (i.e. one that maps to classes that were generated in-memory), a "dry" project (one that does not have Java classes specified, only class names) is passed to the DynamicTypeBuilder, passing in an instance of DynamicClassLoader, which does the work of building the in-memory classes:

InputStream metadata = ...
Project dynamicProject = DynamicTypeBuilder.loadDynamicProject(inputStream, null, new DynamicClassLoader(Thread.currentThread().getContextClassLoader()));

When this project is used to unmarshall XML documents, the objects that are returned by EclipseLink will be subclasses of DynamicEntity. DynamicEntities offer a simple get(propertyName) / set(propertyName, propertyValue) API to manipulate their data:

DynamicEntity dynamicEmployee = (DynamicEntity) jaxbUnmarshaller().unmarshal(instanceDoc);
String firstName = dynamicEmployee.get("fname");
DynamicEntity manager = dynamicEmployee.get("manager");
 
...
 
dynamicEmployee.set("lname", "Duggar");
newAddress.set("street", "1001 Duggar Ranch Way");
dynamicEmployee.set("address", newAddress);

DynamicTypes are analogous to Java Classes, whereas DynamicEntities are instances of a DynamicType.

XJC

XJC (XML-Java Compiler) is the executable compiler that is shipped with JAXB in the JDK. Normally it is run from the command line, taking an XML Schema as input and generating JAXB-annotated Java files on disk. There is a developer API for working with XJC in code as well, and we will leverage this to parse the XML Schema and create in-memory representations of the classes, stopping short of actually generating the Java code.

Requirements

1. Enable the user to use JAXB APIs to marshal and unmarshal XML data, without having Java domain classes defined.
2. Enable the user to bootstrap from various metadata sources, including EclipseLink Project XML, EclipseLink External Metadata, and XML Schema.
3. Enable the user to obtain references to DynamicEntities via either Java class names or XML complex type names.
4. Enable the user to add arbitrary properties/mappings at runtime, after initialization has been performed.
5. Ensure that the Dynamic JAXB workflow is thread-safe.

Design Constraints

Milestones

Milestone 1

• Creation of new DynamicJAXBContext, DynamicJAXBContextFactory DONE
• Creating New DynamicEntites by Java Name DONE
  • API to create new DynamicEntites DONE
  • Code to create new DynamicEntities DONE
• Bootstrapping from Project
  • API to create context from Project object(s) / Deployment XML / sessions.xml DONE
  • Code to obtain a DynamicProject from one or more regular Projects DONE
  • Code to obtain a DynamicProject from one or more Project XMLs DONE
  • Code to create a DynamicProject from sessions.xml session names DONE
  • Test Cases to ensure that ALL mapping types are functioning DONE
    • Using Java names DONE
    • Using XML names

Milestone 2

• Bootstrapping from XJC
  • API to create context from XML Schema DONE
  • Code to parse XJC's class model to create our own JavaModel & pass to MappingsGenerator DONE
  • Code to obtain a DynamicProject from one or more URLs DONE
  • Code to obtain a DynamicProject from one or more Nodes DONE
  • Code to obtain a DynamicProject from one or more InputStreams DONE
  • Code to obtain a DynamicProject from one or more Sources
  • Code to obtain a DynamicProject from one or more XMLStreamReaders
  • Code to obtain a DynamicProject from one or more XMLEventReaders
  • Code to obtain a DynamicProject from one or more resource names
  • Test Cases to ensure that ALL mapping types and annotations are functioning
    • Using Java names
    • Using XML names

Milestone 3

• Bootstrapping from OXM
  • API to create context from OXM DONE
  • Code to parse OXM model to create our own JavaModel & pass to MappingsGenerator
  • Code to obtain a DynamicProject from one or more URLs
  • Code to obtain a DynamicProject from one or more resource names
  • Test Cases to ensure that ALL mapping types and annotations are functioning

Milestone 4

• Creating New DynamicEntites by XML Name (XPath)
• Getting / Setting values by XML Name (XPath)

Future

• Adding Mappings Dynamically

API

DynamicJAXBContextFactory

The entry point to this feature is a new class, DynamicJAXBContextFactory, which is used to create new instances of DynamicJAXBContext. DynamicJAXBContext is a subclass of the existing EclipseLink JAXBContext (which is itself a subclass of the JAXB spec's JAXBContext). The following static methods can be used to create new instances of DynamicJAXBContext from various metadata sources:

//import org.w3c.dom.Node;
//import java.io.InputStream;
//import javax.xml.transform.Source;
 
// XSD Metadata
public static DynamicJAXBContext createContext(Node schemaDOM, SchemaResolver resolver, ClassLoader classLoader, Map<String, ?> properties);
public static DynamicJAXBContext createContext(InputStream schemaStream, SchemaResolverresolver, ClassLoader classLoader, Map<String, ?> properties);
public static DynamicJAXBContext createContext(Source schemaSource, SchemaResolverresolver, ClassLoader classLoader, Map<String, ?> properties);
public static DynamicJAXBContext createContext(String schemaResourceNames, SchemaResolverresolver, ClassLoader classLoader, Map<String, ?> properties);
 
// EclipseLink Project/OXM Metadata
public static DynamicJAXBContext createContext(String sessionNames, ClassLoader classLoader, Map<String, ?> properties);

The ClassLoader parameter is the application's current class loader, and will be used to first lookup classes to see if they exist before new DynamicTypes are generated (i.e., it will become the DynamicClassLoader's parent class loader). The user can pass in null for this parameter, and Thread.currentThread().getContextClassLoader() will be used instead.

When creating a DynamicJAXBContextFactory from EclipseLink Projects, you can supply a colon-delimited String list of EclipseLink Sessions XML session names, e.g.:

String contextPaths = "org.example.customer:org.example.hr";
DynamicJAXBContext dContext = DynamicJAXBContextFactory.createContext(contextPaths, null, null);
// sessions.xml will be searched for sessions named "org.example.customer" and "org.example.hr".
// DynamicTypes for both projects will be available in this DynamicJAXBContext.

A DynamicJAXBContext can also be instantiated by using the newInstance() method on javax.xml.bind.JAXBContext. In order to do this, a jaxb.properties must be present in the specified packages, and it must be configured to use DynamicJAXBContextFactory as the factory class.

// jaxb.properties
 
javax.xml.bind.context.factory=org.eclipse.persistence.jaxb.DynamicJAXBContextFactory

DynamicJAXBContext jaxbContext = (DynamicJAXBContext) JAXBContext.newInstance("org.example.customer:org.example.hr");

Extended JAXBContext Functionality

In addition to the standard JAXBContext APIs, there are several utility methods available on DynamicJAXBContext to obtain type information and create new DynamicEntities (using both Java and XML identifiers).

public DynamicType getDynamicType(String javaName)
public DynamicType getDynamicType(String namespaceURI, String localName)
 
public <T> T newDynamicEntity(Object object, String xPath, NamespaceResolver namespaceResolver, Class<T> returnType)
public <T> T newDynamicEntity(String namespaceURI, String localName, Class<T> returnType)
public <T> T newDynamicEntity(String javaName, Class<T> returnType)
public <T> T newDynamicEntity(DynamicType dynamicType, Class<T> returnType)
 
// From org.eclipse.persistence.jaxb.JAXBContext:
public <T> T getValueByXPath(Object object, String xPath, NamespaceResolver namespaceResolver, Class<T> returnType)
public void setValueByXPath(Object object, String xPath, NamespaceResolver namespaceResolver, Object value)

Using Java Names vs. XML Names

To interact with DynamicEntities, the user has the option of using either Java or XML identifiers. When using Java identifiers, the user can interact with the DynamicEntities directly:

DynamicEntity employee = (DynamicEntity) dContext.createUnmarshaller().unmarshal(new File("instance.xml"));
String lastName = employee.get("lastName");
employee.set("lastName", lastName + " Jr.");
 
DynamicEntity address = dContext.newDynamicEntity("org.example.Address");
address.set("street", "123 Main Street");
 
employee.set("address", address);

To use XML identifiers, the following methods on DynamicJAXBContext are used:

DynamicEntity employee = (DynamicEntity) dContext.createUnmarshaller().unmarshal(new File("instance.xml"));
String lastName = dContext.getValueByXPath(employee, "last-name/text()", aNamespaceResolver, String.class);
dContext.setValueByXPath(employee, "last-name/text()", aNamespaceResolver, lastName + " Jr.");
 
DynamicEntity address = dContext.newDynamicEntity("http://www.example.org", "AddressType");
dContext.setValueByXPath(address, "street/text()", aNamespaceResolver, "123 Main Street");
 
dContext.setValueByXPath(employee, "address", aNamespaceResolver, address);

Creating a DynamicJAXBContext

There are three ways to bootstrap a DynamicJAXBContext; from an EclipseLink Project, EclipseLink External Metadata, or XML Schema:

// From EclipseLink Project
ClassLoader loader = Thread.currentThread().getContextClassLoader();
DynamicJAXBContext dContext = DynamicJAXBContextFactory.createContext("session-name", loader, null);

// From EclipseLink OXM
ClassLoader loader = Thread.currentThread().getContextClassLoader();
DynamicJAXBContext dContext = DynamicJAXBContextFactory.createContext("oxm-name", loader, null);

// From Schema
InputStream inputStream = ClassLoader.getSystemResourceAsStream("resource/xsd/employee.xsd");
 
NamespaceResolver nsResolver = new NamespaceResolver();
nsResolver.put("ns0", "myNamespace");
nsResolver.put("xsi", XMLConstants.SCHEMA_INSTANCE_URL);
 
ClassLoader loader = Thread.currentThread().getContextClassLoader();
DynamicJAXBContext dContext = DynamicJAXBContextFactory.createContext(inputStream, nsResolver, loader, null);

Unmarshalling and Modifying an Object from XML

DynamicEntity employee = (DynamicEntity) dContext.createUnmarshaller().unmarshal(new File("instance.xml"));
employee.set("empId", "Smith");
employee.get("address").set("street", "1001 Riverside Dr.");

Marshalling a New Object

Since, from the user's perspective, no concrete Java classes actually exist, the user must have a way to create new instances. This is done using the newDynamicEntity* methods on DynamicJAXBContext. For example:

Using Java class/field names:

DynamicEntity employee = dContext.newDynamicEntity("org.acme.Employee");
employee.set("firstName", "Bob");
employee.set("lastName", "Barker");
dContext.createMarshaller().marshal(employee, System.out);

Using XML names:

DynamicEntity employee = dContext.newDynamicEntity("http://www.acme.org", "employee-type");
dContext.setValueByXPath(employee, "first-name/text()", aNamespaceResolver, "Bob");
dContext.setValueByXPath(employee, "last-name/text()", aNamespaceResolver, "Barker");
dContext.createMarshaller().marshal(employee, System.out);

Type Lookup

If reference to an actual DynamicType object is desired, it can be obtained by doing a lookup on DynamicJAXBContext, or from a DynamicEntity instance:

DynamicType employeeType = dContext.getDynamicType("org.acme.Employee");
DynamicEntity employee = employeeType.newDynamicEntity();

DynamicType employeeType = dContext.getDynamicType("http://www.acme.org", "employee-type");
DynamicEntity employee = employeeType.newDynamicEntity();

DynamicEntity employee = (DynamicEntity) dContext.createUnmarshaller().unmarshal(new File("instance.xml"));
DynamicType employeeType = employee.getDynamicType();

Examples

Bootstrapping from XML Schema, using XML names:

// Create a DynamicJAXBContext from an XML schema file:
InputStream inputStream = ClassLoader.getSystemResourceAsStream("resource/xsd/employee.xsd");
ClassLoader loader = Thread.currentThread().getContextClassLoader();
 
NamespaceResolver nsResolver = new NamespaceResolver();
nsResolver.put("ns0", "myNamespace");
nsResolver.put("xsi", XMLConstants.SCHEMA_INSTANCE_URL);
 
DynamicJAXBContext dContext = DynamicJAXBContextFactory.createContext(inputStream, nsResolver, loader, null); 
 
// Unmarshal an XML instance doc to get an Employee object:
DynamicEntity employee = (DynamicEntity) dContext.createUnmarshaller().unmarshal(new File("instance.xml"));
String firstName = dContext.getValueByXPath(employee, "first-name/text()", nsResolver, String.class);
String lastName = dContext.getValueByXPath(employee, "last-name/text()", nsResolver, String.class);
String currentEmployeeName = firstName + " " + lastName;
 
// Create a new Address and set it on the Employee:
DynamicEntity newAddress = dContext.getHelper().newDynamicEntity("http://www.acme.org", "address-type");
dContext.setValueByXPath(newAddress, "street/text()", nsResolver, "773 Bank St.");
dContext.setValueByXPath(employee, "address", nsResolver, newAddress);
 
// Marshal the modified Employee:
dContext.createMarshaller().marshal(employee, System.out);

Bootstrapping from EclipseLink Project, using Java names:

// Create a DynamicJAXBContext from an XML schema file:
ClassLoader loader = Thread.currentThread().getContextClassLoader();
 
DynamicJAXBContext dContext = DynamicJAXBContextFactory.createContext("org.acme", loader, null);
 
// Unmarshal an XML instance doc to get an Employee object:
DynamicEntity employee = (DynamicEntity) dContext.createUnmarshaller().unmarshal(new File("instance.xml"));
String currentEmployeeName = employee.get("fName") + " " + employee.get("lName");
 
// Create a new Address and set it on the Employee:
DynamicEntity newAddress = dContext.newDynamicEntity("org.acme.Address");
newAddress.set("street", "773 Bank St.");
employee.set("address", newAddress);
 
// Marshal the modified Employee:
dContext.createMarshaller().marshal(employee, System.out);

Design / Functionality

Bootstrapping from Deployment XML

When constructing a DynamicJAXBContext from Deployment XML, a DynamicClassLoader is first created using the supplied ClassLoader as its parent. An InputStream is then created from the metadata URL and used to create a Project mapped to Dynamic Entities. Finally, XMLContext and DynamicHelper objects are instantiated and stored.

DynamicClassLoader dynamicClassLoader;
if (classLoader instanceof DynamicClassLoader) {
   dynamicClassLoader = (DynamicClassLoader) classLoader;
} else {
   dynamicClassLoader = new DynamicClassLoader(classLoader);            
}
InputStream inputStream = metadataURL.openStream();
Project p = DynamicTypeBuilder.loadDynamicProject(inputStream, null, dynamicClassLoader);
 
this.xmlContext = new XMLContext(p);
this.dynamicHelper = new DynamicHelper(xmlContext.getSession(0));

Bootstrapping from XML Schema

See EclipseLink/Development/2.1/DynamicMOXy/296967/BootstrapFromXSD

Bootstrapping from EclipseLink External Metadata

TBD

Adding Properties/Mappings at Runtime

One of the main aspects of this feature is the ability to add new mappings to an existing running system (aligning with JPA's support for this scenario).

• Mappings must be fully initialized before they are added to the descriptor, for thread-safety.
• Mapping customizations will be isolated to the current DynamicJAXBContext, i.e. changes made to a descriptor will only be visible in that descriptor's XMLContext, even if multiple DynamicJAXBContexts were created from the same initial metadata.

Testing

GUI

No GUI is required for this feature.

Config Files

A new metadata file may be needed to specify and persist new mapping information. (See Open Issues below)

Documentation

Open Issues

This section lists the open issues that are still pending that must be decided prior to fully implementing this project's requirements. (Bold Issue # indicates unresolved issue)

Thread Safety

In a multi-threaded environment, the user is expected to use a single instance of DynamicJAXBContext, shared by client threads (as with standard JAXB). Each thread should create its own Marshallers and Unmarshallers. Consider the following example:

User A:

DynamicEntity employee = (DynamicEntity) dContext.createUnmarshaller().unmarshal(new File("instance1.xml"));
employee.set("first-name", "Bob");
// Some lengthy processing here...
dContext.createMarshaller().marshal(employee, System.out);

User B:

DynamicEntity employee = (DynamicEntity) dContext.createUnmarshaller().unmarshal(new File("instance2.xml"));
dContext.addDirectMapping("empId", String.class, "employee-id", employee.getDynamicType());
employee.set("employee-id", "77264");
dContext.createMarshaller().marshal(employee, System.out);

If User B's code completes before User A's, will User A see "employee-id" in their XML?

Option 1: User A does not see "employee-id" in their XML because it was not present in the DynamicType at the time of unmarshalling.
Option 2: User A does see "employee-id" in their XML, with an empty or null value.
Option 3: User A could have the option of refreshing the DynamicType to see if anything has changed, e.g.:

User A:

DynamicEntity employee = (DynamicEntity) dContext.createUnmarshaller().unmarshal(new File("instance1.xml"));
employee.set("first-name", "Bob");
DynamicType currentEmployeeType = employee.getDynamicType();
DynamicType refreshedEmployeeType = dContext.refreshType(new QName("http://www.acme.org", "employee-type"));
if (currentEmployeeType.getNumberOfProperties() == refreshedEmployeeType.getNumberOfProperties()) {
   dContext.createMarshaller().marshal(employee, System.out);
} else {
   throw new Exception("Type has been modified.");
}

Regardless, users will always have access to the most up to date DynamicType when they call DynamicJAXBContext.createMarshaller()/createUnmarshaller().

Decisions

This section lists decisions made. These are intended to document the resolution of open issues or constraints added to the project that are important.

Future Considerations

During the research for this project the following items were identified as out of scope but are captured here as potential future enhancements. If agreed upon during the review process these should be logged in the bug system.

Appendix

Review Meeting Minutes - Jan 26

Review: Design Specification: MOXy Support for Dynamic Persistence
Tuesday January 26 2010
Attendees: Rick Barkhouse, David Twelves, Doug Clarke, Shaun Smith, Blaise Doughan, Mike Norman, Polly Chang

Minutes:

API / DynamicJAXBContextFactory

• Create names can be shorter, omit 'Context'
• XSD is an acccepted abbreviation for XML Schema
• We will refer to External Metadata as OXM (possibly subject to change)
• createFromXSD(), createFromOXM(), createFromProject()
• Minimize number of createFromOXM and createFromProject APIs, focus on and maximize XSD constructors
• Should add XSD constructors for STAX, and add EntityResolvers to all XSD constructors

Q. Would it be better for users to go through standard JAXB API, rather than creating new API?

• This may be an alternative route for users to access new functionality - could be an enhancement after core feature is complete.
• Would require 'jaxb.properties' on classpath, which may be an issue for users that just have a schema and nothing on filesystem at all (currently don't require a jaxb.properties file with proposed API)
• Could also support bootstrapping via the (String contextPath, Map properties) APIs (future)

Problem encountered by JPA: If asked to create a Static Class that doesn't exist, hard to know whether to go ahead and create a dynamic class or not, or if it is just an error. JPA just treats this as an error and doesn't create dynamic class (correct?) This should be raised as a new open issue.

Should we have a flag in OXM.xml to explicitly indicate that classes should be generated and not loaded?

API / Use Cases

Polly raised usecase of DynaBeans (sp?) where an XPath mapping is stored in the D/B. We would handle this as a separate ORM mapping to the D/B schema and create JAXBContext from there. We have a solution to this at present - would be worth creating an example to capture how we do this.

She also raised another usecase - she has statically defined objects but needs to map extra properties (in addition to the defined mappings) but doesn't know what these will be until runtime.

This is similar to the SDO concept of defined and open content properties, where open content is defined later. Enhancement request: We may want to create an additional core OXM mapping feature to handle this scenario for JAXB as well.

API / ClassLoader parameter

Currently default to default classloader - parameter is there to allow a different classloader to be defined (OSGi usecase requires this)

Do we want another method without the classloader parameter? No, just leave as is and pass in null, want to avoid creating too much API.

Also discussed whether StreamReader and EventReader should be included and whether these are commonly used. Could wait to see if users raise requests for these and add in API later.

Examples

No need for QNames in method parameters, use (String ns, String name) instead.

Would be nice to have the ability to perform sets on DynamicEntities using XPaths

• enhancement request for base DynamicEntity code (Mike)
• could be achieved today using Blaise's new XPath querying API in XMLContext

XMLExample: No namespace qualification. Need default namespace and ability to specify it. This could be an incremental enhancement.

Open Issues

002 Concurrency/Threading

• Dynamic persistence API allows mapping to be added to a descriptor on the fly.
• If entity already exists, it will lazily add a slot for the new mapping (that will be null)
• New version of the schema created on the fly. Generate new dynamic context from new schema and start using that. In-flight requests will continue to use old schema/old dynamic content.
• Can't remove mappings on the fly - a new context would need to be created.

006 Dynamic entities and normal concrete Java classes used together?

• Nice to have enhancement, post 2.1
• JPA - right now is either all dynamic or all static, hybrid model not officially supported.

008 Helper Object?

Might be a good idea to move this API to separate Helper object. Main thing is to be consistent with SDO and JPA.

Feature Priority

• Creating dynamic entities from Project is higher priority than creating from a Schema.
• An XML-names approach seems to be more favored over Java names.