Difference between revisions of "SMILA/Documentation"

Revision as of 07:39, 21 October 2014

Basics

• Installing and Running
• HowTos
• Architecture
• Overview of Default Configuration
• Data Model, XML, JSON, BON
• Using the REST API, REST Client
• REST API Reference
• Enabling Remote Access to SMILA

Development Environment

• Setting up your Eclipse IDE for SMILA
• Building SMILA
• Creating new Components
• Testing new Components
• Adding Third Party Libraries to SMILA
• Using OSGi Declarative Services
• Extending the build process:
  • Adding a new Component Bundle to the Build
  • Adding a new Test Bundle to the Build

Pipelines and Pipelets: Synchronous Workflows

• What are Pipelines? What are Pipelets?
• Configuring and Creating BPEL Pipelines
  • Accessing SMILA records in BPEL activities
• Using the SMILA BPEL Designer
• ReST APIs
  • Getting Pipelet Descriptions
  • Getting Pipeline Descriptions
  • Creating, Editing, and Executing Pipelines
• Basic Pipelets
  • Common Pipelets
  • XML Processing Pipelets
  • Boilerpipe Pipelet (extract text from HTML content)
  • TikaPipelet (extract text from binary content)
  • JdbcLoggingPipelet, JdbcFetcherPipelet, JdbcSelectPipelet (write/read to/from a database)
  • More special pipelets are provided by the components described below.
• Developing new Pipelets
  • Writing a Pipelet
  • Using the Blackboard Service

Scripting

• Scripting SMILA with Javascript

Searching

• Search Processing and APIs
• Solr Integration: Configuration and Pipelets

JobManager: Asynchronous Workflows

• What are Jobs and Tasks?
  • JobManager Walk-Through
• Creating Workflows and Jobs
  • Data Object Types and Buckets
  • Workers and Workflows
  • Jobs
  • Job Parameters
• Running and Monitoring Jobs
• Configuring the Job Manager
• TaskManager: Asynchronous Scheduling of Tasks
• Worker Reference
  • Bulkbuilder worker
  • PipelineProcessor Worker
  • PipeletProcessor Worker
  • (see Importing section for more workers)
• Developing new Workers
  • WorkerManager: Workers Made Easily
  • How to Write a Worker
  • Task Generators

Importing

• Concepts, Workflow and Components
  • Compound Extractor Service
• Reference of Import Workers
  • FileCrawler and FileFetcher Worker
  • WebCrawler and WebFetcher Worker
    • Crawling multiple start URLs in one job run
  • JdbcCrawler and JdbcFetcher Worker
  • FeedCrawler Worker
  • DeltaChecker Worker
  • UpdatePusher Worker
• Developing new Import Workers
  • Using the VisitedLinks service
  • Extending the WebCrawler worker
  • Adding a Data Source to the SMILA Import Framework
• Additionally
  • Remote Crawling

Embedded HTTP Server

• Configuring Jetty
• Developing JSON ReST Handlers for SMILA

Common Services

• Configuration Helper
• Workspace Helper
• ClusterConfig Service
  • Simple Implementation
• Zookeeper Service
• ObjectStore
  • Filesystem Objectstore Implementation
• BinaryStorage
• Processing Security Information
  • Implementation for using LDAP
  • Example Pipelets
• Ontology Processing with Sesame: Configuration and Pipelets
• MimeTypeIdentifier
• Description of Worker and Pipelet Parameters
• Publishing Web Services
• General JPA Configuration in SMILA
• SMILA Version Information
• Adding JDBC Drivers

Deprecated Components

• • JMX Management
    • JMX Clients
  • RecordStorage