Difference between revisions of "SMILA/Manual"

Revision as of 10:20, 19 January 2012

Basics

• Architecture
• Data Model, XML, JSON, BON
• Using the ReST API
• REST API Reference

The SMILA Development Environment

• Setup your Eclipse IDE for SMILA
• Build SMILA
• Creating new components
• Testing new components
• Adding Third Party libraries to SMILA
• Extending the build process:
  • Adding a new component bundle to the build
  • Adding a new test bundle to the build

Pipelines and Pipelets: Synchronous Workflows of Java components

• What are Pipelets?
• Configuration and Creating BPEL Pipelines
  • Accessing SMILA records in BPEL activities
• Using the SMILA BPEL Designer
• Basic Pipelets
  • Common pipelets: bundle org.eclipse.smila.processing.pipelets
  • Common XML pipelets: Bundle org.eclipse.smila.processing.pipelets.xmlprocessing
  • More special pipelets are provided by components described below.
• Developing Pipelets
  • How to write a pipelet
  • BlackboardService

Using SMILA for Search

• Search Processing and APIs
• Solr Integration: Configuration and Pipelets

JobManager: Asynchronous Workflows

• Concepts: Jobs and Tasks
  • A simple example
• Creating Workflows and Jobs
  • Defining Buckets
  • Modeling Workflows
  • Creating Jobs
• Running and Monitoring Jobs
• JobManager Configuration
• TaskManager: Asynchronous scheduling of tasks
• Workers Reference
  • Bulkbuilder
  • PipelineProcesor
  • PipeletProcessor
  • See SMILA/Manual#Importing below for more workers
• Developing Workers
• WorkerManager: Workers made easy
• Developing Workers

Importing

• Concepts
• Importing Workers reference
  • FileCrawler and FileFetcher worker
  • WebCrawler and WebFetcher worker
  • DeltaCheck worker
  • UpdatePusher worker
• Developing Importing Workers
  • VisitedLinks service
  • Extending the WebCrawler worker

The SMILA HTTP Server

• Configuration of Jetty
• Developing JSON ReST handlers for SMILA

Common Services

• Configuration Helper
• Workspace Helper
• ClusterConfig service
  • Simple implementation
• ObjectStore
  • Filesystem Objectstore implementation
• BinaryStorage
• Processing security information
  • Implementation for using LDAP
  • Example Pipelets
• Ontology processing with Sesame
• MimeTypeIdentifier
• Publishing Web Services

Deprecated Components

• Connectivity Framework
  • ConnectivityManager
  • DeltaIndexingManager
  • ConnectivityFramework
    • CrawlerController
    • Crawler
      • Filesystem Crawler
      • Web Crawler
      • JDBC Crawler
    • AgentController
    • Agent
      • Mock Agent
      • Feed Agent
      • JobFile Agent
    • CompoundManagement
  • JMX Management
    • JMX Clients
  • RecordStorage