SMILA/Specifications/CompoundManagementDiscussion

From Eclipsepedia

Jump to: navigation, search

Contents

CompoundManagement improvements

The current CompoundManagement implementation is by no means finished and final. Below are some ideas and already known issues that could or even have to be adressed in the future:

Integration in DeltaIndexingManager

Priority
SHOWSTOPPER
STATUS
DONE see https://bugs.eclipse.org/bugs/show_bug.cgi?id=278360

The current implementations of the DeltaIndexingManager do not handle compound elements correctly. We have to store dependencies between compound records and their elements. If a compound record is checked for update and DeltaIndexing determines that it needs no update then all elements of the compound record have to be marked as visited as well as the compound record. This has to be done recursively for nested compounds (zip in zip in ...).

We have to store two more information with the DeltaIndexingManager:

  • boolean isCompound: a flag that specifies if an entry is a compound record (true) or not (false)
  • String parentIdHash: the hash of the parentId. This is only set for compound elements that reference their direct parent compound record. For top level compounds or non compound records it is set to NULL.

To speed up DeltaIndexing we should not set the VISITED flag for all elements of a compound record, but only for the ones that are containers themselves. In this way we can save lots of modifications on existing entries (especially useful for JPA implementation). We also need an additional flag MODIFIED to differentiate between unchanged and changed compound objects. Therefore we have to spend some more logic when determining the records for DeltaIndexing Delete. Here we have to select only those records whose visited flag is false and that either don't have a parentId (they are not part of a compound hierachy) or a parentId who's MODIFIED flag is set to true.


Adapting Compound Records

Priority
LOW
STATUS
OPEN

CompoundManager offers method adaptCompoundRecord(...) to adapt the compound record after it's elements were extracted. This is useful for the following scenarios:

  • if we do not want SMILA to process and index the compound records themselves we could delete the record
  • if we want to index the compound record (its metadata, the content makes no sense for any search engine) we can do so but we may want to remove the big content object before sending it to the workflow engine
  • anything else ...

At the moment this method is not implemented, it returns the unmodified record. Of course the adaptation should be configurable. Both of the above described options should be easy to implement.

Compound Inheritance

Priority
LOW
STATUS
OPEN

It should be possible to "inherit" attributes, attachments and annotations from a compound record to it's elements. A good example are access rights that are associated with the compound record but are lost when the elements are processed. The inheritance should be configurable:

  • what attributes/attachments/annotations are inherited
  • how are they inherited (execution mode)
    • add: the compound record values are added to existing element values
    • replace: the elements values (if any exist) are replaced by the compound records values
    • setIfEmpty: the values from the compound record are set on the element if no values exist


Filtering of compound elements

Priority
LOW
STATUS
OPEN

It should be possible to configure filters for compound elements so that certain elements of a compound record are ignored just as within regular crawlers. It would be great if the filters of the data source the compound record originates from could be reused but I guess that Crawlers/Agents configuration may get to heterogeneous. So a separate filter mechanism could be applied that works only on the common defined CompoundAttributes (those are anyway the only available attributes).


CompoundHandling for Agents

Priority
LOW
STATUS
OPEN

At the moment Compoundhandling is only used in the CrawlerController (to be precise in the class CrawlThread). It should also be available in the AgentController processing logic although we currently do not have an Agent that provides compound records. Perhaps we can enhance the mock agent to send compound records if so desired to allow testing.

Dependency to MimeTypeIdentifier

Priority
HIGH
STATUS
OPEN

The CompoundManager needs a MimeTypeIdentifier service to be able to identify the mime type of an incoming object and to decide whether it's a compound or not. This already works fine. However, the MimeTypeidentifier interface and the SimpleMimeTypeIdentifier service are located in bundle org.eclipse.smila.processing.pipelets.mimetype which entails dependencies to org.eclipse.smila.processing and some of its sub-bundles. We should move the MimeTypeIdentifier interface and the SimpleMimeTypeIdentifier into different packages outside of processing. Perhaps utils is a good place, but we have to separate interface and implementation to allow for other implementations (ApertureMimetypeidentifier will definitely come). Then the SimpleMimeTypeIdentifier should also be separated into a pure service and a ProcessingService. The ProcessingService should be located in org.eclipse.smila.processing.pipelets and it should be independent of the MimeTypeIdentifier service used. It should work with any MimeTypeIdentifier and contain appropriate logic to find mime type and/or extension information about the files (e.g. the web crawler metadata).

Tutorial

Priority
HIGH
STATUS
OPEN

We should add a Tutorial on "How to implement a CompoundHandler and CompoundCrawler" as it is a common place for contributors to extend SMILA with their own functionality.