Difference between revisions of "SMILA/Documentation"

Revision as of 03:55, 15 June 2012

Basics

• Installing and Running
• Architecture
• Overview of Default Configuration
• Data Model, XML, JSON, BON
• Using the ReST API
• REST API Reference
• Enabling Remote Access to SMILA

Development Environment

• Setting up your Eclipse IDE for SMILA
• Building SMILA
• Creating new Components
• Testing new Components
• Adding Third Party Libraries to SMILA
• Using OSGi Declarative Services
• Extending the build process:
  • Adding a new Component Bundle to the Build
  • Adding a new Test Bundle to the Build

Pipelines and Pipelets: Synchronous Workflows

• What are Pipelines? What are Pipelets?
• Configuring and Creating BPEL Pipelines
  • Accessing SMILA records in BPEL activities
• Using the SMILA BPEL Designer
• ReST APIs
  • Getting Pipelet Descriptions
  • Getting Pipeline Descriptions
  • Creating, Editing, and Executing Pipelines
• Basic Pipelets
  • Common Pipelets in Bundle org.eclipse.smila.processing.pipelets
  • XML Processing Pipelets in Bundle org.eclipse.smila.processing.pipelets.xmlprocessing
  • Boilerpipe Pipelets in Bundle org.eclipse.smila.processing.pipelets.boilerpipe
  • More special pipelets are provided by the components described below.
• Developing new Pipelets
  • Writing a Pipelet
  • Using the Blackboard Service

Searching

• Search Processing and APIs
• Solr Integration: Configuration and Pipelets

JobManager: Asynchronous Workflows

• What are Jobs and Tasks?
  • JobManager Walk-Through
• Creating Workflows and Jobs
  • Defining Buckets
  • Modeling Workflows
  • Creating Jobs
  • Evaluating Job Parameters
• Running and Monitoring Jobs
• Configuring the Job Manager
• TaskManager: Asynchronous Scheduling of Tasks
• Worker Reference
  • Bulkbuilder worker
  • PipelineProcesor Worker
  • PipeletProcessor Worker
  • See Importing below for more workers
• Developing new Workers
  • WorkerManager: Workers Made Easily
  • How to Write a Worker
  • Task Generators

Importing

• Concepts, Workflow and Components
  • Compound Extractor Service
• Reference of Import Workers
  • FileCrawler and FileFetcher Worker
  • WebCrawler and WebFetcher Worker
  • JdbcCrawler and JdbcFetcher Worker
  • DeltaChecker Worker
  • UpdatePusher Worker
• Developing new Import Workers
  • Using the VisitedLinks service
  • Extending the WebCrawler worker
  • Adding a Data Source to the SMILA Import Framework

Embedded HTTP Server

• Configuring Jetty
• Developing JSON ReST Handlers for SMILA

Common Services

• Configuration Helper
• Workspace Helper
• ClusterConfig Service
  • Simple Implementation
• Zookeeper Service
• ObjectStore
  • Filesystem Objectstore Implementation
• BinaryStorage
• Processing Security Information
  • Implementation for using LDAP
  • Example Pipelets
• Ontology Processing with Sesame: Configuration and Pipelets
• MimeTypeIdentifier
• Description of Worker and Pipelet Parameters
• Publishing Web Services
• General JPA Configuration in SMILA

Deprecated Components

• Connectivity Framework
  • ConnectivityManager
  • DeltaIndexingManager
    • CrawlerController
    • Crawler
      • Filesystem Crawler
      • Web Crawler
      • JDBC Crawler
    • AgentController
    • Agent
      • Mock Agent
      • Feed Agent
      • JobFile Agent
    • CompoundManagement
  • Development
    • How to implement a crawler
    • How to implement an agent
  • JMX Management
    • JMX Clients
  • RecordStorage