Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "SMILA/Documentation"
< SMILA
m |
m (→Pipelines and Pipelets: Synchronous Workflows) |
||
| (69 intermediate revisions by 11 users not shown) | |||
| Line 1: | Line 1: | ||
| − | + | == Basics == | |
| − | * | + | * [[SMILA/Documentation_for_5_Minutes_to_Success|Installing and Running]] |
| − | *[ | + | * [[SMILA/Documentation/HowTo|HowTos]] |
| + | * [[SMILA/Documentation/Architecture_Overview|Architecture]] | ||
| + | * [[SMILA/Documentation/Default_configuration_workflow_overview|Overview of Default Configuration]] | ||
| + | * [[SMILA/Documentation/Data_Model_and_Serialization_Formats|Data Model, XML, JSON, BON]] | ||
| + | * [[SMILA/Documentation/Using_The_ReST_API|Using the REST API, REST Client]] | ||
| + | * [[SMILA/Documentation/REST_API_Reference|REST API Reference]] | ||
| + | * [[SMILA/Documentation/Enable Remote Access|Enabling Remote Access to SMILA]] | ||
| − | + | == Development Environment == | |
| − | + | * [[SMILA/Documentation/HowTo/Howto_set_up_dev_environment|Setting up your Eclipse IDE for SMILA]] | |
| − | + | * [[SMILA/Documentation/HowTo/Howto_build_a_SMILA-Distribution|Building SMILA]] | |
| − | + | * [[SMILA/Documentation/HowTo/Create_a_bundle_(plug-in)|Creating new Components]] | |
| − | + | * [[SMILA/Documentation/HowTo/Create_a_test_bundle_(plug-in)|Testing new Components]] | |
| − | + | * [[SMILA/Documentation/HowTo/3rd_party_bundle|Adding Third Party Libraries to SMILA]] | |
| − | + | * [[SMILA/Documentation/Declarative_Services|Using OSGi Declarative Services]] | |
| − | **[[SMILA/Documentation/ | + | * Extending the build process: |
| − | **[[SMILA/Documentation/ | + | ** [[SMILA/Documentation/HowTo/How_to_integrate_new_bundle_into_build_process|Adding a new Component Bundle to the Build]] |
| + | ** [[SMILA/Documentation/HowTo/How_to_integrate_test_bundle_into_build_process|Adding a new Test Bundle to the Build]] | ||
| − | + | == Pipelines and Pipelets: Synchronous Workflows == | |
| − | + | * [[SMILA/Documentation/Pipelets|What are Pipelines? What are Pipelets?]] | |
| − | + | * [[SMILA/Documentation/BPEL_Workflow_Processor|Configuring and Creating BPEL Pipelines]] | |
| − | **[[SMILA/Documentation/ | + | ** [[SMILA/Documentation/HowTo/How_to_filter_and_access_record_data_in_BPEL|Accessing SMILA records in BPEL activities]] |
| − | + | * [[SMILA/BPEL_Designer|Using the SMILA BPEL Designer]] | |
| − | + | * ReST APIs | |
| − | + | ** [[SMILA/Documentation/Processing/JSON_REST_API_for_pipelets|Getting Pipelet Descriptions]] | |
| − | + | ** [[SMILA/Documentation/Processing/JSON_REST_API_for_BPEL_pipelines|Getting Pipeline Descriptions]] | |
| − | * | + | ** [[SMILA/Documentation/Processing/JSON_REST_API_for_BPEL_pipelines|Creating, Editing, and Executing Pipelines]] |
| − | + | * Basic Pipelets | |
| − | + | ** [[SMILA/Documentation/Bundle org.eclipse.smila.processing.pipelets|Common Pipelets]] | |
| − | + | ** [[SMILA/Documentation/Bundle org.eclipse.smila.processing.pipelets.xmlprocessing|XML Processing Pipelets]] | |
| − | + | ** [[SMILA/Documentation/Bundle org.eclipse.smila.processing.pipelets.boilerpipe|Boilerpipe Pipelet (extract text from HTML content)]] | |
| − | + | ** [[SMILA/Documentation/TikaPipelet|TikaPipelet (extract text from binary content)]] | |
| − | + | ** [[SMILA/Documentation/JdbcLoggingPipelet|JdbcLoggingPipelet, JdbcFetcherPipelet, JdbcSelectPipelet (write/read to/from a database)]] | |
| − | + | ** More special pipelets are provided by the components described below. | |
| − | + | * Developing new Pipelets | |
| − | + | ** [[SMILA/Documentation/HowTo/How_to_write_a_Pipelet|Writing a Pipelet]] | |
| − | **[[SMILA/Documentation/Processing/ | + | ** [[SMILA/Documentation/Usage_of_Blackboard_Service|Using the Blackboard Service]] |
| − | * | + | |
| − | **[[SMILA/Documentation/ | + | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | *** | + | |
| − | + | ||
| − | + | ||
| − | + | == Searching == | |
| − | + | ||
| − | *[[SMILA/Documentation/ | + | * [[SMILA/Documentation/Search|Search Processing and APIs]] |
| − | + | * [[SMILA/Documentation/Solr 3.5|Solr Integration: Configuration and Pipelets]] | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | * | + | == JobManager: Asynchronous Workflows == |
| − | **[[SMILA/Documentation/ | + | * [[SMILA/Documentation/JobManager|What are Jobs and Tasks?]] |
| − | **[[SMILA/Documentation/ | + | ** [[SMILA/Documentation/JobManagerFirstExample|JobManager Walk-Through]] |
| − | **[[SMILA/Documentation/ | + | * Creating Workflows and Jobs |
| − | ** | + | ** [[SMILA/Documentation/DataObjectTypesAndBuckets|Data Object Types and Buckets]] |
| − | ***[[SMILA/Documentation/Bulkbuilder|Bulkbuilder worker]] | + | ** [[SMILA/Documentation/WorkerAndWorkflows|Workers and Workflows]] |
| − | ***[[SMILA/Documentation/Worker/ | + | ** [[SMILA/Documentation/JobDefinitions|Jobs]] |
| − | ***[[SMILA/Documentation/Worker/ | + | ** [[SMILA/Documentation/JobParameters|Job Parameters]] |
| + | * [[SMILA/Documentation/JobRuns|Running and Monitoring Jobs]] | ||
| + | * [[SMILA/Documentation/JobManagerConfiguration|Configuring the Job Manager]] | ||
| + | * [[SMILA/Documentation/TaskManager|TaskManager: Asynchronous Scheduling of Tasks]] | ||
| + | * Worker Reference | ||
| + | ** [[SMILA/Documentation/Bulkbuilder|Bulkbuilder worker]] | ||
| + | ** [[SMILA/Documentation/Worker/PipelineProcessorWorker|PipelineProcessor Worker]] | ||
| + | ** [[SMILA/Documentation/Worker/PipeletProcessorWorker|PipeletProcessor Worker]] | ||
| + | ** (see [[SMILA/Manual#Importing|Importing]] section for more workers) | ||
| + | * Developing new Workers | ||
| + | ** [[SMILA/Documentation/WorkerManager|WorkerManager: Workers Made Easily]] | ||
| + | ** [[SMILA/Documentation/HowTo/How_to_write_a_Worker|How to Write a Worker]] | ||
| + | ** [[SMILA/Documentation/TaskGenerators|Task Generators]] | ||
| − | *[[SMILA/Documentation/ | + | == Importing == |
| − | + | * [[SMILA/Documentation/Importing/Concept|Concepts, Workflow and Components]] | |
| − | * | + | ** [[SMILA/Documentation/Importing/CompoundExtractorService|Compound Extractor Service]] |
| − | **[[SMILA/Documentation/ | + | * Reference of Import Workers |
| − | **[[SMILA/Documentation/ | + | **[[SMILA/Documentation/Importing/Crawler/File | FileCrawler and FileFetcher Worker]] |
| − | ** | + | **[[SMILA/Documentation/Importing/Crawler/Web | WebCrawler and WebFetcher Worker]] |
| − | + | *** [[SMILA/Documentation/Importing/CrawlingMultipleStartURLs | Crawling multiple start URLs in one job run]] | |
| − | * | + | **[[SMILA/Documentation/Importing/Crawler/JDBC | JdbcCrawler and JdbcFetcher Worker]] |
| − | **[[SMILA/Documentation/ | + | **[[SMILA/Documentation/Importing/Crawler/Feed | FeedCrawler Worker]] |
| + | **[[SMILA/Documentation/Importing/DeltaCheck | DeltaChecker Worker]] | ||
| + | **[[SMILA/Documentation/Importing/UpdatePusher | UpdatePusher Worker]] | ||
| + | * Developing new Import Workers | ||
| + | ** [[SMILA/Documentation/Importing/VisitedLinks | Using the VisitedLinks service]] | ||
| + | ** [[SMILA/Documentation/Importing/Crawler/Web#Internal_structure|Extending the WebCrawler worker]] | ||
| + | ** [[SMILA/Documentation/HowTo/How to add a new Data Source to the importing framework|Adding a Data Source to the SMILA Import Framework]] | ||
| + | * Additionally | ||
| + | ** [[SMILA/Documentation/Importing/RemoteCrawling | Remote Crawling]] | ||
| − | * | + | == Embedded HTTP Server == |
| − | **[[SMILA/Documentation/ | + | * [[SMILA/Documentation/JettyHttpServer|Configuring Jetty]] |
| + | * [[SMILA/Documentation/JettyHttpServer#JSON_Handlers|Developing JSON ReST Handlers for SMILA]] | ||
| + | |||
| + | == Common Services == | ||
| + | * [[SMILA/Project Concepts/Simple configuration handler|Configuration Helper]] | ||
| + | * [[SMILA/Documentation/Workspace Handling|Workspace Helper]] | ||
| + | * [[SMILA/Documentation/Bundle_org.eclipse.smila.clusterconfig|ClusterConfig Service]] | ||
| + | ** [[SMILA/Documentation/Bundle_org.eclipse.smila.clusterconfig.simple|Simple Implementation]] | ||
| + | * [[SMILA/Documentation/Bundle_org.eclipse.smila.zookeeper|Zookeeper Service]] | ||
| + | * [[SMILA/Documentation/ObjectStore/Bundle_org.eclipse.smila.objectstore|ObjectStore]] | ||
| + | ** [[SMILA/Documentation/ObjectStore/Bundle_org.eclipse.smila.objectstore.filesystem|Filesystem Objectstore Implementation]] | ||
| + | * [[SMILA/Documentation/Binary_Storage|BinaryStorage]] | ||
| + | * [[SMILA/Documentation/Security|Processing Security Information]] | ||
| + | ** [[SMILA/Documentation/LDAPSecurityResolver|Implementation for using LDAP]] | ||
| + | ** [[SMILA/Documentation/SecurityProcessingServices|Example Pipelets]] | ||
| + | * [[SMILA/Documentation/SesameOntologyManager|Ontology Processing with Sesame: Configuration and Pipelets]] | ||
| + | * [[SMILA/Documentation/MimeTypeIdentifier|MimeTypeIdentifier]] | ||
| + | * [[SMILA/Documentation/ParameterDefinition|Description of Worker and Pipelet Parameters]] | ||
| + | * [[SMILA/Documentation/PublishingJAXWSWebservices|Publishing Web Services]] | ||
| + | * [[SMILA/Documentation/General JPA Configuration in SMILA|General JPA Configuration in SMILA]] | ||
| + | * [[SMILA/Documentation/SMILA_Versioning|SMILA Version Information]] | ||
| + | * [[SMILA/Documentation/Adding_JDBC_Drivers|Adding JDBC Drivers]] | ||
| + | |||
| + | == Deprecated Components == | ||
| + | |||
| + | ** [[SMILA/Documentation/Management|JMX Management]] | ||
| + | *** [[SMILA/Documentation/Management#JMX_Client|JMX Clients]] | ||
| + | ** [[SMILA/Documentation/Record_Storage|RecordStorage]] | ||
[[Category:SMILA]] | [[Category:SMILA]] | ||
Revision as of 09:52, 12 August 2014
Contents
Basics
- Installing and Running
- HowTos
- Architecture
- Overview of Default Configuration
- Data Model, XML, JSON, BON
- Using the REST API, REST Client
- REST API Reference
- Enabling Remote Access to SMILA
Development Environment
- Setting up your Eclipse IDE for SMILA
- Building SMILA
- Creating new Components
- Testing new Components
- Adding Third Party Libraries to SMILA
- Using OSGi Declarative Services
- Extending the build process:
Pipelines and Pipelets: Synchronous Workflows
- What are Pipelines? What are Pipelets?
- Configuring and Creating BPEL Pipelines
- Using the SMILA BPEL Designer
- ReST APIs
- Basic Pipelets
- Common Pipelets
- XML Processing Pipelets
- Boilerpipe Pipelet (extract text from HTML content)
- TikaPipelet (extract text from binary content)
- JdbcLoggingPipelet, JdbcFetcherPipelet, JdbcSelectPipelet (write/read to/from a database)
- More special pipelets are provided by the components described below.
- Developing new Pipelets
Searching
JobManager: Asynchronous Workflows
- What are Jobs and Tasks?
- Creating Workflows and Jobs
- Running and Monitoring Jobs
- Configuring the Job Manager
- TaskManager: Asynchronous Scheduling of Tasks
- Worker Reference
- Bulkbuilder worker
- PipelineProcessor Worker
- PipeletProcessor Worker
- (see Importing section for more workers)
- Developing new Workers
Importing
- Concepts, Workflow and Components
- Reference of Import Workers
- Developing new Import Workers
- Additionally
Embedded HTTP Server
Common Services
- Configuration Helper
- Workspace Helper
- ClusterConfig Service
- Zookeeper Service
- ObjectStore
- BinaryStorage
- Processing Security Information
- Ontology Processing with Sesame: Configuration and Pipelets
- MimeTypeIdentifier
- Description of Worker and Pipelet Parameters
- Publishing Web Services
- General JPA Configuration in SMILA
- SMILA Version Information
- Adding JDBC Drivers