Difference between revisions of "SMILA/Documentation"

Revision as of 12:44, 24 January 2012

Basics

• Architecture
• Overview on Default Configuration
• Data Model, XML, JSON, BON
• Using the ReST API
• REST API Reference

The SMILA Development Environment

• Setup your Eclipse IDE for SMILA
• Build SMILA
• Creating new components
• Testing new components
• Adding Third Party libraries to SMILA
• Using OSGi Declarative Services
• Extending the build process:
  • Adding a new component bundle to the build
  • Adding a new test bundle to the build

Pipelines and Pipelets: Synchronous Workflows of Java components

• What are Pipelines? What are Pipelets?
• Configuration and Creating BPEL Pipelines
  • Accessing SMILA records in BPEL activities
• Using the SMILA BPEL Designer
• ReST APIs
  • for showing pipelets
  • for showing, manipulating and executing pipelines
• Basic Pipelets
  • Common pipelets: bundle org.eclipse.smila.processing.pipelets
  • Common XML pipelets: Bundle org.eclipse.smila.processing.pipelets.xmlprocessing
  • More special pipelets are provided by components described below.
• Developing Pipelets
  • How to write a pipelet
  • BlackboardService

Using SMILA for Search

• Search Processing and APIs
• Solr Integration: Configuration and Pipelets

JobManager: Asynchronous Workflows

• Concepts: Jobs and Tasks
  • A simple example
• Creating Workflows and Jobs
  • Defining Buckets
  • Modeling Workflows
  • Creating Jobs
  • Evaluation of Job Parameters
• Running and Monitoring Jobs
• JobManager Configuration
• TaskManager: Asynchronous scheduling of tasks
• Workers Reference
  • Bulkbuilder
  • PipelineProcesor
  • PipeletProcessor
  • See Importing below for more workers
• Developing Workers
  • WorkerManager: Workers made easy
  • How to write a worker
  • Task Generators

Importing

• Concepts
• Importing Workers reference
  • FileCrawler and FileFetcher worker
  • WebCrawler and WebFetcher worker
  • DeltaCheck worker
  • UpdatePusher worker
• Developing Importing Workers
  • VisitedLinks service
  • Extending the WebCrawler worker
  • How to add a new Data Source to the SMILA importing framework

The SMILA HTTP Server

• Configuration of Jetty
• Developing JSON ReST handlers for SMILA

Common Services

• Configuration Helper
• Workspace Helper
• ClusterConfig service
  • Simple implementation
• ObjectStore
  • Filesystem Objectstore implementation
• BinaryStorage
• Processing security information
  • Implementation for using LDAP
  • Example Pipelets
• Ontology processing with Sesame: Configuration and Pipelets
• MimeTypeIdentifier
• Publishing Web Services
• General JPA Configuration in SMILA

Deprecated Components

• Connectivity Framework
  • ConnectivityManager
  • DeltaIndexingManager
    • CrawlerController
    • Crawler
      • Filesystem Crawler
      • Web Crawler
      • JDBC Crawler
    • AgentController
    • Agent
      • Mock Agent
      • Feed Agent
      • JobFile Agent
    • CompoundManagement
  • Development
    • How to implement a crawler
    • How to implement an agent
  • JMX Management
    • JMX Clients
  • RecordStorage