Difference between revisions of "SMILA/Manual"

Revision as of 10:16, 19 January 2012

The SMILA Development Environment

• Setup your Eclipse IDE for SMILA
• Build SMILA
• Creating new components
• Testing new components
• Adding Third Party libraries to SMILA
• Extending the build process:
  • Adding a new component bundle to the build
  • Adding a new test bundle to the build

Basics

• Architecture
• Data Model, XML, JSON, BON
• Using the ReST API
• REST API Reference

Pipelines and Pipelets: Synchronous Workflows of Java components

• What are Pipelets?
• Configuration and Creating BPEL Pipelines
  • Using SMILA records in BPEL activities
• Using the SMILA BPEL Designer|
• Basic Pipelets
  • Common pipelets: bundle org.eclipse.smila.processing.pipelets
  • Common XML pipelets: Bundle org.eclipse.smila.processing.pipelets.xmlprocessing
  • More special pipelets are provided by components described below.
• Developing Pipelets
  • How to write a pipelet?
  • BlackboardService

Using SMILA for Search

• Search Processing and APIs
• Solr Integration Pipelets

JobManager: Asynchronous Workflows

• Concepts: Jobs and Tasks
  • A simple example
• Creating Workflows and Jobs
  • Defining Buckets
  • Modeling Workflows
  • Creating Jobs
• Running and Monitoring Jobs
• JobManager Configuration
• TaskManager: Asynchronous scheduling of tasks
• Workers Reference
  • Bulkbuilder
  • PipelineProcesor
  • PipeletProcessor
  • See SMILA/Manual#Importing below for more workers
• Developing Workers
• WorkerManager: Workers made easy
• Developing Workers

Importing

• • Concepts: Crawler, Fetcher, Delta, VisitedLinks
  • Importing Workers reference
    • FileCrawler worker
    • WebCrawler worker
    • DeltaCheck worker
    • UpdatePusher worker
  • Developing Importing Workers
    • VisitedLinks service
    • Extending the WebCrawler worker

The SMILA HTTP Server

• Configuration of Jetty
• Developing JSON ReST handlers for SMILA

Common Services

• Configuration Helper
• Workspace Helper
• ClusterConfig service
  • Simple implementation
• ObjectStore
  • Filesystem Objectstore implementation
• SMILA/Documentation/Binary_Storage
• Processing security information
  • SMILA/Documentation/LDAPSecurityResolver
  • SMILA/Documentation/SecurityProcessingServices
• Ontology processing with Sesame
• MimeTypeIdentifier
• Publishing Web Services

Deprecated Components

• Connectivity Framework
  • ConnectivityManager
  • DeltaIndexingManager
  • ConnectivityFramework
    • CrawlerController
    • Crawler
      • Filesystem Crawler
      • Web Crawler
      • JDBC Crawler
    • AgentController
    • Agent
      • Mock Agent
      • Feed Agent
      • JobFile Agent
    • CompoundManagement
  • JMX Management
    • JMX Clients
  • RecordStorage