View source for EclipseLink/DesignDocs/362899

<div style="margin:5px;float:right;border:1px solid #000000;padding:5px">__TOC__</div>
= Design Specification: Support NoSQL Databases =

[https://bugs.eclipse.org/bugs/show_bug.cgi?id=362899 ER 362899]

[[Talk:EclipseLink/DesignDocs/362899 | Feedback]]

= Document History =
{|{{BMTableStyle}}
|-{{BMTHStyle}}
! Date
! Author
! Version Description & Notes
|- 
| 2011-12-08
| James
| 0.1 Draft
|- 
|}

= Project overview =
There are several "NoSQL" databases that are non-relational.
These databases include:
* Oracle NoSQL
* MongoDB
* Hadoop
* Google BigTable
* Cassandra

These databases offer alternative APIs to JDBC and either do not support querying, or support alternative query languages to SQL.

You could also include legacy hierarchical and non-relational databases, object databases and XML databases in the "NoSQL" category.

It is desirable for EclipseLink to support persistence and specifically JPA access to NoSQL databases.

EclipseLink currently provides persistence support for relational databases through JDBC, and non-relational datasources through JCA.
This support requires a JDBC or JCA resource adapter/driver for the database.
Most NoSQL databases do not provide a standard driver, although some may, and some do provide JDBC drivers.
There are also third parties that provide JDBC and JCA driver to a wide range of datasources, including some NoSQL databases.

See:
* [http://code.google.com/a/apache-extras.org/p/cassandra-jdbc/ Apache Cassandra JDBC Driver]
* [https://github.com/erh/mongo-jdbc Mongo JDBC Driver]
* [http://www.datadirect.com/supported_data_sources/index.html DataDirect]
* [http://www.attunity.com/attunity_connect_application_adapters Attunity JCA] [http://www.attunity.com/attunity_connect_jdbc_drivers JDBC]

Users connecting EclipseLink to NoSQL databases have several options:
* If they have a JDBC driver available they can connect using EclipseLink's relational support, creating their own DatabasePlatform in EclipseLink will provide enhanced functionality and ease of use.
* If they have a JCA driver available they can connect using EclipseLink's EIS support, creating their own EISPlatform in EclipseLink will provide enhanced functionality and ease of use.
* They can implement their own JCA (or JDBC) adapter for their database to enable usage of EclipseLink's EIS support.

EclipseLink currently provides several JCA adapters to non-relational datasources.
These include:
* JMS
* Oracle AQ
* XML file adapter

EclipseLink should provide a platform and adapter to popular NoSQL databases, similar to the platforms provided for popular relational databases.

EclipseLink's EIS support currently has several limitations that must be resolved to support NoSQL databases.
* No JPA annotation or XML support is currently available for EIS descriptors, mappings, and queries.
* Although hooks exists, no current EIS adapters provide Expression (JPQL/Criteria) translation into non-SQL query languages or APIs.
* The JPA API has not been tested with EIS descriptors and datasources.

= Concepts =
* NoSQL - see [http://en.wikipedia.org/wiki/NoSQL NoSQL - Wikipedia]
* Hierarchical data - data that is nested like XML, not flat like relational data, most NoSQL data is hierarchical.
* EIS - Enterprise Information System, includes non-relational databases, gateways, transaction monitors, and applications. (VSAM, ADA, CICS, IMS, SAP, Salesforce, MongoDB, BigTable).

= Requirements =
* Support JPA access to EIS and NoSQL databases.
* Support NoSQL JPA annotations and XML.
* Support JPQL subset (superset?) for NoSQL databases that support querying.
* Support mapping XML/JSON data from NoSQL databases.
* Support EISPlatforms and adapters for key NoSQL databases:
** Oracle NoSQL
** MongoDB
** others?

= Design Constraints =
* Some NoSQL databases do not support querying.
* Transaction semantics may be different, or even not exist in some NoSQL databases.
* Data and its access is non-standard and may require different types of mappings, queries or data access.
* NoSQL databases may be schema free, and may support storing of dynamic data.
* NoSQL databases provide non standard Java APIs, our adapters will require compile dependencies with these APIs.

= Functionality =
Support NoSQL annotations and XML in JPA.
* @NoSql annotation will allow a class to be tagged as mapping NoSQL data.
* JPA @Basic, @Embedded, @ElementCollection, @OneToOne, @ManyToOne, @OneToMany, @ManyToMany will be allowed for NoSQL classes and mapped to EIS mappings.
* @Field annotation will allow defining the data field name, similar to @Column, (@Column will also be supported)
* Future - @NamedInteractionQuery will allow a native query to be defined using an EIS interaction
* Future - A subset of JAXB and Moxy annotations will be allowed, @XmlAttribute, @XmlElement, @XmlPath in place of @Column (to start with).
** Other JAXB annotations my be supported, or added in the future.
** Unsupported JAXB annotations will be checked for and log warnings or trigger errors.

Support JPA auto-mapping and defaulting.
* Defaulting rules will be similar to relational.  Some JAXB defaulting rules may also be used.

Support JCA adapter and EISPlatform for Oracle NoSQL.
* GET, PUT, PUT_IF_ABSENT, PUT_IF_PRESENT, PUT_IF_VERSION, DELETE, DELETE_IF_VERSION, ITERATOR operations will be supported.
* keyed XML data will be supported.
* nested key/value data will be supported using major and minor keys.
* Oracle NoSQL does not support querying, so JPQL, Criteria querying will not be supported, only CRUD (find) and native interaction based queries will be supported.
** JPQL or Criteria for all objects, or by id will be supported.
* Oracle NoSQL has no transaction support, the JPA transaction API will be supported, but flushes will be committed and aborts will not rollback committed changes.
* Future - Oracle NoSQL has some support for versioning, it may be possible to support optimistic locking.
** Oracle NoSQL versining is complex and not a good fit to JPA versioning.
* Support for query options, CONSISTENCY, DURABILITY, TIMEOUT, VERSION will be supported.
* Sequencing is not support by Oracle NoSQL, but will be supported in EclipseLink through UUID
** Support for UUIDSequence and JPA UuidGenerator support will be added, @GeneratedValue will default to UUID.

Support JCA adapter and EISPlatform for MongoDB.
* BSON data mapped through MappedRecords.
* INSERT, UPDATE, REMOVE, FIND operations will be supported.
* A subset of JPQL and Criteria (through Expressions) will be translated to Mongo BSON based querying:
** =, <>, <, >, <=, >=
** like (translated to regex)
** in, not in
** and, or, not
** traversals to composite object fields
** order by of fields asc/desc
** select of fields (no aggregate functions or group by)
* Mongo also supports some level of advanced querying that may be supported:
** Future - update alls, delete alls
** Future - group support using Java script
** Future - mod is supported, other functions are not supported
** Future - regex operations are supported, but have no JPQL equivalent
** joins will not be supported, joins to element collections may be possible
** sub-selects will not be supported
** parallel selects will not be supported
* Future - Mongo has mapReduce functions that allows executing JavaScript in the database, not sure how/if that maps to persistence
* Future - Mongo has a native command string that could be mapped to native queries.
* Mongo has some support for versioning, optimistic locking will be supported
* Mongo has support for generated ids through its oid support, oid will be used as the default id generator.
* Mongo has no transaction support, the JPA transaction API will be supported, but flushes will be committed and aborts will not rollback committed changes.
* Future - Mongo has support for schema indexes and creating dropping collections, this could be supported through JPA's ddl/schema options.
* Query options, READ_PREFERENCE, WRITE_CONCERN, OPTIONS will be supported.
* JPA/core query options, maxResults, firstResult, fetchSize will be supported through SKIP, LIMIT, BATCH_SIZE.

Future - Support for map keys may be possible.

Future - Composite collection mappings do no currently support change tracking, support may be desirable.

= Testing =
Testing will require access and installation to supported NoSQL databases.

Testing will be done locally by committers responsible for the NoSQL database's integration (i.e. Oracle).

JPA and native API test models will be provided for Oracle NoSQL and MongoDB.

Oracle NoSQL will test:
* XML data
** GenerateValue
** UuidGenerator
** Basic
** Embedded
** OneToOne
** orm.xml
* Mapped data
** Basic
** Embedded
** OneToOne
** orm.xml
* CRUD operations
* Native interaction queries

MongoDB will test:
* Mapped data
** GenerateValue
** Version
** Basic
** Embedded
** ElementCollection
** OneToOne
** TODO - OneToMany
* CRUD operations
* JPQL queries
* Native interaction queries
* TODO - Composite/SessionBroker with mixed relational data

= API =
* @NoSql - Tags descriptor as NoSQL
* @Field - synonym for @Column
* @JoinField, @JoinFields synonyms for @JoinColumn, @JoinColumns
* Future - JAXB - @XmlAttribute, @XmlElement
* Future - Moxy - @XmlPath
* Future - @NamedInteractionQuery

= Native API =
* OracleNoSQLPlatform
* OracleNoSQLConnectionSpec
* MongoPlatform
* MongoConnectionSpec

= Examples =
There are three possibilities in defining the JPA metadata.
* 1 - Use and extend JPA annotations/XML, and ignore/log warnings when incorrect artifacts are used
** Pros: Does not require new annotations/xml, simplest, leverages users JPA expertise/document/tooling
** Cons: Need to validate that relational artifacts are not used, XML schema will allow mixed content (will need to validate XML is not using mixed content)
* 2 - Mirror the JPA annotations/xml in EIS annotations/xml constructs
** Pros: Can better restrict what artifacts are supported (although annotations are not very restrictive in general, XML can be better restricted)
** Cons: Need to define Eis mirrors to most JPA annotations, is confusing and more difficult to learn new configuration
* 3 - Use the JAXB annotations/xml
** Pros: Can leverage existing JAXB artifacts
** Cons: Confusing using JAXB and JPA annotations together, some overlap in JAXB and JPA is confusing which to use, large set of functionality to support
* 4 - Hybrid
** Pros: Probably the best solution
*** Use JPA annotations where applicable
*** Support @Field, @JoinField in place of @Column to avoid relational constructs
*** Support minimal set of applicable JAXB annotations when mapping XML data, (XmlAttribute, XmlElement, XmlPath)
*** Do not use Moxy XML, use JPA XML metadata with <xml-element> instead of <xml-column>
*** Use warnings and validation to handle cases where relational/eis/xml constructs are incorrectly specified


Option #1 - JPA annotations
<source lang="java">
@Entity
@Eis
public class Employee {
  @Id
  @Basic
  long id;

  @Basic
  @Convert("dateConverter")
  Calendar hireDate;

  @Embedded
  @Column(name="addr")
  Address address;

  @ElementCollection
  @Column(name="phone")
  List<Phone> phones;

  @OneToOne
  @JoinColumn(name="mgr")
  Employee manager;
}
</source>

<source lang="java">
<entity name="Employee" class="org.acme.Employee" access="FIELD">
  <eis/>
  <attributes>
    <id name="id"/>
    <basic name="hireDate">
      <convert name="dateConverter"/>
    </basic>
    <embedded name="address">
      <column name="addr"/>
    </embedded>
    <element-collection name="phones">
      <column name="phone"/>
    </element-collection>
    <one-to-one name="phones">
      <join-column name="mgr"/>
    </one-to-one>
  </attributes>
</entity>
</source>

Option #2 - EIS annotations
<source lang="java">
@Entity
@Eis
public class Employee {
  @Id
  @EisBasic
  long id;

  @EisBasic
  @Convert("dateConverter")
  Calendar hireDate;

  @EisEmbedded
  @EisField(name="addr")
  Address address;

  @EisElementCollection
  @EisField(name="phone")
  List<Phone> phones;

  @OneToOne
  @EisJoinField(name="mgr")
  Employee manager;
}
</source>

<source lang="java">
<entity name="Employee" class="org.acme.Employee" access="FIELD">
  <eis/>
  <attributes>
    <eis-id name="id"/>
    <eis-basic name="hireDate">
      <convert name="dateConverter"/>
    </eis-basic>
    <eis-embedded name="address">
      <eis-field name="addr"/>
    </eis-embedded>
    <eis-element-collection name="phones">
      <eis-field name="phone"/>
    </eis-element-collection>
    <eis-one-to-one name="phones">
      <eis-join-field name="mgr"/>
    </eis-one-to-one>
  </attributes>
</entity>
</source>

Option #3 - JAXB annotations
<source lang="java">
@Entity
@Eis
public class Employee {
  @XmlId
  @XmlAttribute
  long id;

  @XmlAttribute
  @XmlJavaTypeAdapter(DateConverter.class)
  Calendar hireDate;

  @XmlElement(name="addr")
  Address address;

  @XmlElement(name="phone")
  List<Phone> phones;

  @XmlIDREF
  Employee manager;
}
</source>

<source lang="java">
<entity name="Employee" class="org.acme.Employee" access="FIELD">
  <eis/>
  <attributes>
    <xml-attribute java-attribute-name="id" xml-id="true"/>
    <xml-attribute java-attribute-name="hireDate">
      <xml-adapter class="DateConverter"/>
    </xml-attribute>
    <xml-element name="addr" java-attribute-name="address"/>
    <xml-element name="phone" java-attribute-name="phones">
    <xml-element name="mgr" java-attribute-name="phones" xml-idref="true"/>
  </attributes>
</entity>
</source>

Option #4 - Hybrid
<source lang="java">
@Entity
@Eis(dataFormat=XML)
public class Employee {
  @Id
  @Basic
  @XmlAttribute
  long id;

  @Basic
  @XmlAttribute
  @Convert("dateConverter")
  Calendar hireDate;

  @Embedded
  @XmlElement(name="addr")
  Address address;

  @ElementCollection
  @XmlElement(name="phone")
  List<Phone> phones;

  @OneToOne
  @XmlElement(name="mgr")
  Employee manager;
}
</source>

<source lang="java">
<entity name="Employee" class="org.acme.Employee" access="FIELD">
  <eis/>
  <attributes>
    <id name="id">
      <xml-attribute/>
    </id>
    <basic name="hireDate">
      <xml-attribute/>
      <convert name="dateConverter"/>
    </basic>
    <embedded name="address">
      <xml-element name="addr"/>
    </embedded>
    <element-collection name="phones">
      <xml-element name="phone"/>
    </element-collection>
    <one-to-one name="phones">
      <xml-element name="mgr"/>
    </one-to-one>
  </attributes>
</entity>
</source>

= Config files =

persistence.xml
* "eclipselink.nosql.connection-spec" - JCA connection spec
* "eclipselink.nosql.property" - JCA driver property (synonymy for "eclipselink.jdbc.property")
* "eclipselink.nosql.platform" - EIS platform (synonymy for "eclipselink.target-database")
* "eclipselink.jdbc.property" - JDBC driver property
* "eclipselink.jdbc.connector" - JDBC connector

= Documentation =
Need a section on NoSQL and EIS data in the JPA user guide.

= Open Issues  =


= Decisions  =

# Oracle NoSQL and MongoDB were decided for initial investigation.
# @NoSql will be used and NoSQL used in docs/API, EIS will not be used other than existing EIS mappings
# JPA metadata will reuse JPA annotations for mappings, @Field, @JoinField will replace @Column, @JoinColumn
# Oracle NoSQL dependency will be added to Oracle component, others will be part of NoSQL component following similar rules to Oracle component dependencies

= Future Considerations =
* Cassandra
* CouchDB
* Google BigTable
* Redis
* RavenDB
* MemcacheDB
* Hadoop, HBase
* Oracle Coherence (as a datasource instead of a cache)
* Oracle Berkeley DB