Revision as of 09:59, 25 April 2011

Flex Columns Extension

This feature allows mappings to be added to preexisting (unmapped) columns in a table.

The Schema is designed to include preallocated columns that can be used to map additional data. In this example, Customer table might look like this:

• CUSTOMER
  • INTEGER ID
  • VARCHAR NAME
  • VARCHAR FLEX_COL1
  • VARCHAR FLEX_COL2
  • VARCHAR FLEX_CO31

The columns starting with the String "FLEX_COL" are flexible and may have mappings that point to after initial application design. For instance at after the application has been designed, a user may add a mapping called "company" that maps to FLEX_COL1.

The use case for a scenario like this is:

• One company develops an application with the goal of making it available to a customer. The application contains all the functionality necessary for basic use of the application
• The application is deployed and made available to the customer. The customer can then tweak the application by adding mappings to data that is specific to their business

This kind of application works well in concert with a multi-tenant application where the initial application is developed and then each tenant can add mappings that are particular to their business.

Requirements

• Users MUST be able to add mappings after initial deployment
• Extensions MUST be persistent. (i.e. Extensions must continue to exist if an application goes down and comes back up)
• Extensions MUST be shared amoung all EntityManagers configured to use them
• Extensions MUST be compatible with our multi-tenant features. (i.e. Extensions must be able to make use of our Multi-tenant features to be definable on a tenant by tenant basis)
• BasicMappings MUST be supported
• Extensibility MUST be configurable using traditional JPA means (annotations, eclispelink-orm.xml)
• It MUST be possible to use JPA Queries to query based on the extensions
• Extensions SHOULD be supported through the JPA metamodel
• OneToOneMappings SHOULD be supported
• It MAY be possible to add extensions without logging in the session. (See ER 341429.)

Configuration

Metadata

Extensions will be supported using our VIRTUAL access type. Virtual Access type allows properties to be set through a getter that takes an attribute name as an argument and a setter that takes an argument name and a value as an argument.

VIRTUAL access does not currently allow annotation-based configuration. As part of this feature, annotations will be added to indicate that an Entity uses VIRTUAL access. Those annotations will be used to enable Extensions.

The annotations indicating use of VIRTUAL access will be used to determine which methods EclipseLink can weave. This weaving will provide equivalent functionality to weaving for PROPERTY and FIELD access. (e.g. change tracking, fetch groups, etc)

DDL generation for tables with flexible columns will be addressed in a separate feature that addresses flexible DDL generation as a complete feature.

Examples

Example 1

• Field Access for non extension fields
• Virtual Access for extension fields uses defaults (get(String), set(String, Object))
• extensions mapped in a portable way - @Transient

  @Entity
  @VirtualProperties()
  public class Address {
 
    @Id
    private int id;
 
    @Transient
    private Map<String, Object> extensions;
 
    public int getId(){
        return id;
    }
 
    public <T> T get(String name) {
        return (T) extentions.get(name);
    }
 
    public Object set(String name, String value) {
        return extensions.put(name, value);
    }
 
...

Example 2

• Field Access for non extension fields
• extensions mapped in a portable way - @Transient
• @VirtualProperties annotation overrides method to be used for get and for set.
• XML for extended mapping indicates which get and set method to use

  @Entity
  @VirtualProperties(get="getExtension", set="setExtension")
  public class Address {
 
    @Id
    private int id;
 
    @Transient
    private Map<String, Object> extensions;
 
    public int getId(){
        return id;
    }
 
    public <T> T getExtension(String name) {
        return (T) extensions.get(name);
    }
 
    public Object setExtension(String name, String value) {
        return extensions.put(name, value);
    }
 
...

 
   <basic name="name" attribute-type="String">
      <column name="FLEX_1"/>
      <access-methods get-method="getExtension" set-method="setExtension"/>
    </basic>

Example 3

• Property Access for non extension fields
• Virtual Access for extension fields uses defaults (get(String), set(String, Object))
• extensions mapped in a portable way - no @Transient required because of Property access

  @Entity
  @VirtualProperties
  public class Address {
 
    private int id;
 
    private Map<String, Object> extensions;
 
    @Id
    public int getId(){
        return id;
    }
 
    public <T> T get(String name) {
        return (T) extensions.get(name);
    }
 
    public Object set(String name, String value) {
        return extensions.put(name, value);
    }
 
...

EntityManagerFactory and Metadata Repository

Extensions will be added at bootstrap time through access to a metadata repository. A Metadata Repository will accessed through a class that provides methods to retrieve the metadata it holds.

The user will specifiy the class to use and any configuration information for the metadata repository through persistence unit properties. As an EntityManagerFactory bootstraps, if metadata repository information is provided, the EMF will check the metadata repository for additional mapping information and integrate it into the metadata it uses to bootstrap.

EclipseLink will initially ship with the capability of connecting to two types of metadata repository.

1. Database Table - information about extensions is stored in a database table
2. XML - information about extensions is stored in XML

Additionally, the user will be able to provide an implementation of the class that access the metadata repository.

Each metadata repository access class will specify an individual set of properties to use to connect to the repository

Examples

XML File

  <property name="eclipselink.metadata-repository" value="XMLFile"/>
  <property name="eclipselink.metadata-repository.xml-file.url" value="foo://bar"/>

Database

  <property name="eclipselink.metadata-repository" value="Database"/>
  <property name="eclipselink.metadata-repository.database.url" value="jdbc:mysql://localhost/metadata"/>
  <property name="eclipselink.metadata-repository.database.user" value="username"/>
  <property name="eclipselink.metadata-repository.database.password" value="password"/>

Note: additional information such as table name, may be required.

User-Specified

  <property name="eclipselink.metadata-repository" value="com.foo.MetadataRepository"/>
  <property name="com.foo.MetadataRepository.location" value="foo://bar"/>
  <property name="com.foo.MetadataRepository.extra-data" value="foo-bar"/>

Note: The implementer of com.foo.MetadataRepository will be free to choose the properties that their implementation requires.

Design

Configuration

Please use the examples above for intial discussion of configuration XML and annotations. Full specification will be provided in this document after an initial discussion. Note also, that the configuration options to allow a mapping to use VIRTUAL access have been available in EclipseLink for several releases. We will be using those configuration options as they exist and any changes to those will be handled as bugs in that feature rather than through this design document.

Weaving

The intial VIRTUAL access feature did not include weaving of the get and set methods. As part of the extensions feature will will add weaving of get and set methods that use virtual access. The initial implementation will only support Basic mappings and throw an exception at Transformer construction time if weaving is requested for a VIRTUAL mapping that is not Basic.

Get method

Original

   public <T> T get(String name) {
       return (T) getExtensions().get(name);
   }

Weaved

   public Object get(String name)
   {
       _persistence_checkFetched(name);
       return getExtensions().get(name);
   }

Set method

Original

   public Object set(String name, Object value)
   {
       return getExtensions().put(name, value);
   }

Weaved

   public Object set(String name, Object value)
   {
       Object obj = null;
       if(_persistence_listener != null)
       {
           obj = get(name);
       } else
       {
           _persistence_checkFetchedForSet(name);
       }
       _persistence_propertyChange(name, obj, value);
       return getExtensions().put(name, value);
   }

EntityManagerFactory

EntityManagerFactory work is still in the early design phase. The following is provided for discussion new API:

• refresh() - refreshes the underlying session by rebootstrapping.

A mechanism will be provided that allows a user to tell a Metadata repository to refresh. That mechanism will take two forms.

1. A direct refresh API call. This call will clear all session information from the EntityManagerFactory without removing that information from existing EntityManagers and cause the EntityManagerFactory to rebootstrap. The rebootstrap will cause the metadata repository to be consulted and any new mapping information will be retrieved.
2. A RemoteCommandManager command that causes all subscribed EntityManagerFactories to refresh themselves as described above.

EntityManagerFactoryWrapper

Current: EntityManagerFactoryImpl -> ServerSession

New: EntityManagerFactoryWrapper -> EntityManagerFactoryImpl -> ServerSession

In both cases, a live EntityManager holds a reference to EntityManagerFactoryImpl

When a call is made to refresh on the MetadataRepository, refresh will be called on EntityManagerFactoryWrapper. EntityManagerFactoryWrapper will bootstrap a new EntityManagerFactoryImpl and use it as the basis for any new EntityManagers. The old EntityManagerFactoryImpl will continue to be available until the last EntityManager is no longer used, at which point we will rely on garbage collection to clean it up.

Bootstrapping

Bootstrapping using the MetadataRepository will be very similar to the current bootstrapping process. In EntityManagerSetupImpl.predeploy(), the MetadataProcessor builds the session based on the existing XML files. During that proces, the MetadataProcessor will extract the new mapping information from the MetadataRepository by asking for an eclipselink-orm.xml representation of it. That file will be used as an override for other orm.xml files used in bootstrapping process.

Metadata Repository

MetadataRepository is still in the early design phase. The following is provided for discussion

Abstract class MetadataRepository will contain 3 types of abstract API

• API to retrieve extensions
  • XMLEntityMappings getEntityMappings(Properties, ClassLoader) - returns XMLEntityMappings which are then merged in using existing metadata processing at bootstrap time when creating an entityManager
  • java.io.Reader getEntityMappingsReader()- returns an InputStreamReader to an eclipselink-orm.xml document that maps the extensions, which can then be used by getEntityMappings implementations. This may not be used in some MetadataRepository implementations.

• API to store extensions and cause them to be used
  • addMapping(DatabaseMapping, Descriptor)
  • notify() - notify send a message to all known MetadataRepositories to indicate they should refresh. (RemoteCommandManager message)
• Configuration API
  • setLocation(String location) - the location of the store, a URL. This could represent the a file system location where the repository is, a database connection URL, or any other location

DatabaseMetadataRepository will be a subclass of MetadataRepository containing additional configuration API

• setUser(String)
• setPassword(String)
• setDriver(String)
• API to configure which table to access and how the table is configured
• getColumn(columnNamePrefix, tableName, dataType) - get an unused column from the given table that can store dataType and has a name staring with the columnNamePrefix. This will be used by users to build the DatabaseMappings to add.

XMLMetadataRepository will also be a subclass of MetadataRepository providing file-based XML lookup

Old Design

Please ignore the design below. It has been kept around until any useful parts of it can be incorporated in the update design.

FlexExtensionManager

Each ClassDescriptor may have an ExtensionManager. For extensible types, the ExtensionManager will be non-null and will be set either by annotation, xml, or explicitly on the descriptor. Different types of ExtensionManager are possible. The flex column behavior is managed by a FlexExtensionManager.

FlexExtensionManager holds the following state:

• extensionsAccessor - an Accessor that gives it access to the Map on the domain class that holds the extensions. e.g. the Map annotated as @FlexExtensions
• availableFields - the list of DatabaseField that it believes can be used for new extensions
• usedFields - the list of DatabaseField that it believes is used for existing extensions
• shouldCreateNonExistingColumns - a boolean used to enable the additional feature that will allow ALTER schema language to create new columns (more to come)

Extension Data

Extension data will be persisted to the database. Each extension will be represented by an ExtensionProperty holding the following data:

• String - entityType - the name of the Entity it represents
• DatabaseField - field - a DatabaseField representing the column used by this extension
• Class - type - the target class of the mapping
• String - name - the name of the extension
• mapping - a reference to the mapping for this extension

An project containing an Extensible type will have a descriptor added to it for ExtensionProperty. Any time an Extension is added, its ExtensionProperty will be persisted. The table will contain the following information:

• ENTITY_TYPE (PK) - Directly mapped from entityType
• COLUMN_NAME (PK) - Name of the field stored in field
• TARGET_TYPE - Class name of the class represented by type
• NAME - Directly mapped from name

At descriptor initialization time, a ReadAllQuery will be issued for all ExtensionProperties with an ENTITY_TYPE. These will be stored by the ExtensionManager and used to build the appropritate mappings.

Adding Extensions

Extensions are added through a call to extensionManager.addExtension(name, type). When addExtension is called, the following will occur:

1. a check will be made to ensure the extension does not already exist in the descriptor and an exception will be thrown if it does
2. the list of available fields will be checked for a field that can represent the type that is passed in
3. If no field is available either the Column Creation feature will be used, or an exception will be thrown.
4. the availableFields and usedFields lists will be updated
5. An instance of ExtensionProperty will be created and persisted, if persisting fails for any reason, an exception will be thrown and the availableFields and usedFields lists will be reset
6. A mapping will be created to represent the ExtensionProperty and it will be added to the descriptor

Mapping Types

Extensions may only be the following types of mappings

• Basic
• OneToOne (single foreign key)

MapValueAttributeAccessor

Since all extensions will be stored in the same map, mappings cooresponding with those extensions will use a MapValueAttributeAccessor.

MapValueAttributeAccessor will have knowledge of the map holding the extensions and a call to getAttributeValueFromObject will retreive the value from the map based on the extension's property name and a call to setAttributeValueInObject will put a value in the map based on the property name.

Basic Mappings

The initial iteration will allow Basic mappings. Basic mappings are fairly easy to implement as they are a simple coorespondance between a field and value in the map. A basic mapping will be constructed with the appropriate data and added to the descriptor

OneToOne Mappings

OneToOne mappings to Objects with a single primary key will be supported. The flex column will be used as the foreign key.

Extension Propogation

Extensions will be refreshed from the database each time an EntityManagerFactory or EntityManager is created.

No Push style notification via our RemoteCommandManager or other mechanism is currently planned.

DDL Generation

DDL generation will allow the creation of the extension table and also include any known extension columns to domain tables

Column Creation

See <URL for Column Creation doc here>

Future Enhancemts

Virtual Access access the extensions field directly

• Field Access for non extension fields
• Virtual Access for extension fields uses @Extensions annotation on attribute that will hold the extensions. This assumes an enhancement to allow Virtual access to operate on a field
• extensions mapped in a non portable way - @Extensions
• Preexisting table assumed. @FlexExtensions annotation omitted

  @Entity
  @Extensible(FLEX)
  public class Address {
 
    @Id
    private int id;
 
    @Extensions
    private Map<String, Object> extensions;
 
    public int getId(){
        return id;
    }
 
...

Appendices

Apendix 1: Annotations

@Target({TYPE}) 
@Retention(RUNTIME)
public @interface FlexExtensions {
    /**
     * (Optional) Specify number of columns.  This is used to automatically generate a default set of columns. 
     * Unnecessary if columns is specified
     */
    int numberOfColumns default 10;
 
    /**
     * The string to prefix to the name of automatically generated columns.
     * e.g. the first flex column could be "FLEX_COL1" and the second, "FLEX_COL2" etc
     */
    String columnNamePrefix default "FLEX_COL";
 
    /**
     * The string to prefix to the name of automatically generated columns.
     * e.g. the first flex column could be "FLEX_COL1" and the second, "FLEX_COL2" etc
     */
    String defaultColumnType default VARCHAR(255);
 
    /**
     * When set to true this will activate a feature that executes ALTER table statements to 
     * make the required columns available if they do not exist
     */
    boolean createNonExistingColumns default false;
 
    /**
     *  (Optional) The table that holds the extensions to use.  This table can be shared.
     */
    Table table;
 
    /**
     * (Optional) Define column overrides
     */
    FlexColumn[] columns
}
 
@Target({TYPE}) 
@Retention(RUNTIME)
public @interface FlexColumn {
 
    /**
     * Mandatory - the index of the column to override.  Between 1 and numberOfColumns.
     * Exception will be thrown for numbers < 1 or > numberOfColumns
     */
    int index;
 
    /**
     * Column name - used to override default column names
     */
    String name;
 
    /**
     * Column type - used to override default column type
     */
    String type default VARCHAR(255)
}

Appendix 2: eclipselink-orm

  <xsd:complexType name="attributes">
    <xsd:annotation>
      <xsd:documentation>
 
        This element contains the entity field or property mappings.
        It may be sparsely populated to include only a subset of the
        fields or properties. If metadata-complete for the entity is true
        then the remainder of the attributes will be defaulted according
        to the default rules.
 
      </xsd:documentation>
    </xsd:annotation>
    <xsd:sequence>
    ...
    <xsd:element name="flex-extensions" type="orm:flex-extensions" minOccurs="0"/>
 
<!-- **************************************************** -->
 
<xsd:complexType name="flex-extensions">
  <xsd:annotation>
    <xsd:documentation>
      ...
    </xsd:documentation>
  </xsd:annotation>
  <xsd:sequence>
    <xsd:attribute name="number-of-columns" type="xsd:integer"/>
    <xsd:element name="column-name-prefix" type="xsd:string"/>
    <xsd:attribute name="default-column-type" type="xsd:string"/>
    <xsd:attribute name="create-non-existing-columns" type="xsd:boolean"/>
    <xsd:attribute name="extension-table" type="eclipselink-orm:table"/>
    <xsd:element name="flex-columns" type="eclipselink-orm:flex-column" minOccurs="0" maxOccurs="unbounded"/>
  </xsd:sequence>
</xsd:complexType>
 
<xsd:complexType name="flex-column">
  <xsd:annotation>
    <xsd:documentation>
      ...
    </xsd:documentation>
  </xsd:annotation>
  <xsd:sequence>
    <xsd:attribute name="index" type="xsd:integer"/>
    <xsd:attribute name="name" type="xsd:string"/>
    <xsd:attribute name="type" type="xsd:string"/>
  </xsd:sequence>
</xsd:complexType>