Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/Binary Storage"

m (Overview: added category)
Line 1: Line 1:
 +
[[Category:SMILA]]
 +
 
== Overview ==
 
== Overview ==
  

Revision as of 02:56, 17 October 2009


Overview

Binary Storage Service provides an easy way to store / access binary data documents.

Client components access the Binary Storage Service for persisting binary data (attachments) into the binary storage. The binary data are identified by a unique key / identifier as a String data type.

Configuration

Currently, Binary Storage is able to run and store data into a hierarchical structure (uses deterministically calculation based on the hash id passed by the client component, like the ID passed from the Blackboard Service) and flat structure.

To configure the needed persistence storage structure, simply edit the /configuration/org.eclipse.smila.binarystorage/BinaryStorageConfiguration.xml file and change the value of implementationClass attribute to the one of the following available possibilities:

  • org.eclipse.smila.binarystorage.internal.impl.persistence.filesystem.IOHierarchicalManager
  • org.eclipse.smila.binarystorage.internal.impl.persistence.filesystem.IOFlatManager

In the next future there will be other available managers that provides the appropriate implementation for Binary Storage persistence structure.

The Binary Storage Configuration will be changed soon.

Configuration example

<BinaryStorageConfiguration xmlns="http://www.eclipse.org/smila/binarystorage" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:noNamespaceSchemaLocation="schemas/BinaryStorageConfiguration.xsd"
  name="default1"
  provider="file"
  mountPoint="default"
  implementationClass="org.eclipse.smila.binarystorage.internal.impl.persistence.filesystem.IOHierarchicalManager" 
  tempFileName="dummy.dat"
  pathDepth="2"
/>

Attributes explained:

  • name - configuration alias
  • implementationClass - binary storage implementation manager class
  • tempFileName - binary storage temporary file name
  • pathDepth - the depth of the folder structure (Note: If not set the default value is 2.)

API

void store(String id, InputStream stream);
void store(String id, byte[] blob)
byte[] fetchAsByte(String id)
InputStream fetchAsStream(String id)
void remove(String id)
int fetchSize(String id)

API Usage

BinaryStorageService binaryStorageService = ...; // Obtain the binary storage service
String attacmentIdKey = ...; // Unique ID
byte[] dataRecord = ... // Build record
 
// Store the binary record as byte array
binaryStorageService.store(attacmentIdKey, dataRecord);
 
// Fetch record as stream
final InputStream stream = binaryStorageService.fetchAsStream(attacmentIdKey);
 
// Define new ID and store new data from the first record's input stream
final String newAttacmentIdKey = attacmentIdKey + "0000";
binaryStorageService.store(newAttacmentIdKey, stream);
stream.close();
 
// Fetch the newest record as array of byte 
final byte[] recordByte = binaryStorageService.fetchAsByte(newAttacmentIdKey);
 
// The two records must have the same size
final int sizeInitial = _binaryStorageService.fetchSize(attacmentIdKey);
final int sizeFinal = binaryStorageService.fetchSize(newAttacmentIdKey);
 
// Remove the two records
binaryStorageService.remove(attacmentIdKey);
binaryStorageService.remove(newAttacmentIdKey);

Implementations

eclipse.smila.binarystorage.persistence.io

Both I/O implementations use the id in the path name of the persisted file. As a consequence

  • the characters [;/\:] are not allowed in the id
  • too long ids will exceed the platform specific supported max. path length. this will result in a not so obvious error message, that saving failed.

ATM neither limitation this is a problem because the id is a hash made from letters and numbers.

IOHierarchicalManager

The calculation of the path is simply to split the first part of the id into substrings of length 2 and make thees parent folders having the name of the id.

Care must be taken that the folder depth is not too high, because this results in a lot of folders quenching the performance of the file system.

IOFlatManager

If u only have a few files or ur underlying file system is very good inhandling lots of files per folder then go with this manager.


eclipse.smila.binarystorage.persistence.efs

This implementation is based on EFS.


eclipse.smila.binarystorage.persistence.jpa

This implementation uses eclipseLink JPA to store binary objects in an apache derby database named BinaryStorage. The data is stored in table BINARY_OBJECTS:

BINARY_OBJECTS
Column Type Description
ID VARCHAR the Id of the binary object
BIN_OBJECT BLOB the binary object


Configuration

In order to use this implementation you have to start bundle eclipse.smila.binarystorage.persistence.jpa in your launch configuration. Then set parameter implementationClass in file org.eclipse.smila.binarystorage.impl/BinaryStorageConfiguration.xml to org.eclipse.smila.binarystorage.persistence.jpa.JPABinaryPersistence. All other configuration options of file org.eclipse.smila.binarystorage.impl/BinaryStorageConfiguration.xml are ignored !

The other configuration needed is a typicall eclipseLink configuration property file. Therin you can specify settings for logging, database connection settings. For more information please refer to the eclipseLink documentation [[1]]. The configuration is located at configuration/org.eclipse.smila.binarystorage.persistence.jpa/persistence.properties.

# EclipseLink properties
eclipselink.logging.level=INFO
eclipselink.target-server=None
eclipselink.target-database=org.eclipse.persistence.platform.database.DerbyPlatform
eclipselink.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
eclipselink.jdbc.url=jdbc:derby:workspace/.metadata/.plugins/org.eclipse.smila.binarystorage.persistence.jpa/binarystorage;create=true
eclipselink.jdbc.password=smila
eclipselink.jdbc.user=smila
eclipselink.ddl-generation=create-tables

After starting Smila for the first time, the DDL generation setting will print out some nasty warnings, complaining that it can't create some tables. These warnings are not critical. You can get rid of them by setting eclipselink.ddl-generation=none, but only after Smila was started at least once (and the tables were created).

Limitations

At the moment it is necessary to import all packages containing JDBCDriver classes in org.eclipse.smila.connectivity.deltaindexing.jpa.impl. So for changing from derby to another database it is not sufficient to change the configuration in persistence.properties, you also have to add import package statementsv for the JDBC driver to use to your bundles manifest. This will hopefully be changed with the next release of eclipseLink.

Copyright © Eclipse Foundation, Inc. All Rights Reserved.