Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "SMILA/Glossary"
< SMILA
(→B) |
m (→B) |
||
(17 intermediate revisions by the same user not shown) | |||
Line 35: | Line 35: | ||
* '''Action''' - An action is one step in an [[#W|asynchronous workflow]] associated with a certain [[#W|worker]] that does the actual processing. | * '''Action''' - An action is one step in an [[#W|asynchronous workflow]] associated with a certain [[#W|worker]] that does the actual processing. | ||
− | |||
− | |||
* '''Attachment''' - Attachments are parts of [[#R|records]] used to store large binary data such as document content. | * '''Attachment''' - Attachments are parts of [[#R|records]] used to store large binary data such as document content. | ||
− | * '''Attribute''' - Attributes are parts of [[#R|records]] and contain simple data objects that are easily represented in XML, such as <tt>String</tt>, <tt>Integer</tt>, <tt>Float</tt>, and <tt>Date</tt>. | + | * '''Attribute''' - Attributes are parts of [[#R|records]] and contain simple data objects that are easily represented in XML or json, such as <tt>String</tt>, <tt>Integer</tt>, <tt>Float</tt>, and <tt>Date</tt>. |
== B == | == B == | ||
− | * '''Blackboard''' or '''blackboard service''' - The blackboard service | + | * '''Blackboard''' or '''blackboard service''' - The blackboard service holds SMILA [[#R|records]] during processing in a SMILA component. In addition it hides the handling of record persistence from these components. For a complete description see [[SMILA/Documentation/Usage_of_Blackboard_Service|Usage of Blackboard Service]]. |
* '''[http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wsbpel BPEL]''' - BPEL is an XML-based language defining several constructs to write business processes. It defines a set of basic control structures like conditions or loops as well as elements to invoke web services and receive messages from services. It relies on [[#W|WSDL]] to express web services interfaces. Message structures can be manipulated, assigning parts or the whole of them to variables that can in turn be used to send other messages. | * '''[http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wsbpel BPEL]''' - BPEL is an XML-based language defining several constructs to write business processes. It defines a set of basic control structures like conditions or loops as well as elements to invoke web services and receive messages from services. It relies on [[#W|WSDL]] to express web services interfaces. Message structures can be manipulated, assigning parts or the whole of them to variables that can in turn be used to send other messages. | ||
Line 48: | Line 46: | ||
* '''Bucket''' - Data container in an [[#W|asynchronous workflow]], containing logically grouped [[#D|data objects]] of the same type. Can be ''transient'' for interim data, which means that data is not persisted and removal of data is under job management control, or ''persistent'', which means that removal of data is not under job management control. | * '''Bucket''' - Data container in an [[#W|asynchronous workflow]], containing logically grouped [[#D|data objects]] of the same type. Can be ''transient'' for interim data, which means that data is not persisted and removal of data is under job management control, or ''persistent'', which means that removal of data is not under job management control. | ||
− | * '''Bulk''' - a | + | * '''Bulk''' - a number of [[#R|records]] bundled in a single file to enhance throughput when processing records in [[#W|asynchronous workflows]]. |
− | * '''Bulkbuilder''' - An [[#W|asynchronous workflow]] worker that accepts single [[#R|records]] and combines them to a [[#B|bulk]]. See [[SMILA/Documentation/Bulkbuilder|Bulkbuilder documentation]]. | + | * '''Bulkbuilder''' - An [[#W|asynchronous workflow]] [[#W|worker]] that accepts single [[#R|records]] and combines them to a [[#B|bulk]]. See [[SMILA/Documentation/Bulkbuilder|Bulkbuilder documentation]]. |
== C == | == C == | ||
− | * '''Crawler''' - A crawler is a | + | * '''Crawler''' - A crawler is a special [[#W|worker]] in an [[#W|asynchronous workflow]] that imports data from a data source (e.g. filesystem, web or database) into SMILA. It iterates over the data elements and creates [[#R|records]] for all elements that will be further processed in the workflow. In general crawlers resp. crawl workflows are used for initial (bulk) import of data sources. (see SMILA [[SMILA/Documentation#Importing|Importing]] for more details) |
== D == | == D == | ||
Line 60: | Line 58: | ||
* '''Data Object''' - The smallest unit of data handled by an asychronous workflow (e.g. a [[#R|record bulk]]). | * '''Data Object''' - The smallest unit of data handled by an asychronous workflow (e.g. a [[#R|record bulk]]). | ||
− | * ''' | + | * '''DeltaChecker''' - The DeltaChecker is a [[#W|worker]] in an (asynchronous) import [[#W|workflow]] that handles the [[#D|delta indexing]]. |
− | * ''' | + | * '''Delta indexing''' - Delta indexing is also known as incremental or generation based indexing. It is driven by <tt>DeltaChecker</tt> [[#W|worker]].(see SMILA [[SMILA/Documentation#Importing|Importing]] for more details) |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
== E == | == E == | ||
Line 80: | Line 69: | ||
== F == | == F == | ||
+ | |||
+ | * '''Fetcher''' - A fetcher is a [[#W|worker]] in an (asynchronous) import [[#W|workflow]] that retrieves Records containing an URL or file path, etc from a [[#C|crawler]] and actually fetches the content (e.g. of files) from the data source ((e.g. <tt>FileFetcherWorker</tt> or <tt>WebFetcherWorker</tt>)), attaches it to [[#R|records]] and sends them to the [[#U|UpdatePusher]]. (see SMILA [[SMILA/Documentation#Importing|Importing]] for more details) | ||
== G == | == G == | ||
Line 87: | Line 78: | ||
== I == | == I == | ||
− | * '''ID''' - An ID identifies a [[#R|record]] in SMILA and is part of a [[#R|record's]] metadata. | + | * '''ID''' - An ID identifies a [[#R|record]] in SMILA and is part of a [[#R|record's]] metadata. |
− | + | ||
== J == | == J == | ||
− | |||
− | |||
* '''[http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/ JMX]''' - Java Management Extension is a specification to administrating and monitoring java applications. | * '''[http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/ JMX]''' - Java Management Extension is a specification to administrating and monitoring java applications. | ||
Line 98: | Line 86: | ||
* '''Job''' - A Job is a description of a distinct and repeatable working process that the system should accomplish. It references and parametrizes an [[#W|asynchronous workflow]]. | * '''Job''' - A Job is a description of a distinct and repeatable working process that the system should accomplish. It references and parametrizes an [[#W|asynchronous workflow]]. | ||
− | * '''Job Run''' - A Job Run is an "instance" of a Job, for example one run of an import of a data source to an index. | + | * '''Job Run''' - A Job Run is an "instance" of a Job, for example one run of an import of a data source to an index. Only one active job run can existe per job. Statistics will be accumulated for each job run. A job run is automatically stopped when SMILA shuts down. |
== K == | == K == | ||
Line 105: | Line 93: | ||
== M == | == M == | ||
+ | |||
+ | * '''Micro bulk''' - a (small) bundle of [[#R|records]] in one single file which can be pushed into the system using the [[#B|Bulkbuilder]]. The micro bulk in itself is not JSON, but a file where each line must consist of a single JSON representation of a [[#R|record]]. E.g.: | ||
+ | <pre> | ||
+ | {"_recordid": "id1", "attribute1": "attribute1", ...} | ||
+ | {"_recordid": "id2", "attribute1": "attribute2", ...} | ||
+ | {"_recordid": "id3", "attribute1": "attribute3", ...} | ||
+ | </pre> | ||
== N == | == N == | ||
Line 128: | Line 123: | ||
== S == | == S == | ||
− | |||
− | |||
− | |||
− | |||
* '''Slot''' - An (input/output) slot is a description for the input/output behaviour of a [[#W|worker]]. In a concrete [[#W|asynchronous workflow]] slots are assigned to [[#B|buckets]] | * '''Slot''' - An (input/output) slot is a description for the input/output behaviour of a [[#W|worker]]. In a concrete [[#W|asynchronous workflow]] slots are assigned to [[#B|buckets]] | ||
* '''SNMP''' - Simple Network Management Protocol is a network protocol which controls the communication between supervised devices and the monitoring application (e.g. [[#J|JMX]]). | * '''SNMP''' - Simple Network Management Protocol is a network protocol which controls the communication between supervised devices and the monitoring application (e.g. [[#J|JMX]]). | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
== T == | == T == | ||
* '''Task''' - Description of a single unit of work to be processed by a [[#W|Worker]]. A task can contain worker specific properties. | * '''Task''' - Description of a single unit of work to be processed by a [[#W|Worker]]. A task can contain worker specific properties. | ||
+ | * '''[http://tika.apache.org/ Tika]''' - The Apache Tika™ toolkit detects and extracts metadata and structured text content from various documents using existing parser libraries | ||
− | + | == U == | |
− | + | * '''UpdatePusher''' - The UpdatePusher is a [[#W|worker]] in an (asynchronous) import [[#W|workflow]] that pushes the crawled records to the [[#B|BulkBuilder]] of a running import [[#J|job]]. | |
== V == | == V == |
Latest revision as of 03:09, 7 April 2015
A | | B | | C | | D | | E | | F | | G | | H | | I | | J | | K | | L | | M | | N | | O | | P | | Q | | R | | S | | T | | U | | V | | W | | X | | Y | | Z |
---|
A
- Action - An action is one step in an asynchronous workflow associated with a certain worker that does the actual processing.
- Attachment - Attachments are parts of records used to store large binary data such as document content.
- Attribute - Attributes are parts of records and contain simple data objects that are easily represented in XML or json, such as String, Integer, Float, and Date.
B
- Blackboard or blackboard service - The blackboard service holds SMILA records during processing in a SMILA component. In addition it hides the handling of record persistence from these components. For a complete description see Usage of Blackboard Service.
- BPEL - BPEL is an XML-based language defining several constructs to write business processes. It defines a set of basic control structures like conditions or loops as well as elements to invoke web services and receive messages from services. It relies on WSDL to express web services interfaces. Message structures can be manipulated, assigning parts or the whole of them to variables that can in turn be used to send other messages.
- Bucket - Data container in an asynchronous workflow, containing logically grouped data objects of the same type. Can be transient for interim data, which means that data is not persisted and removal of data is under job management control, or persistent, which means that removal of data is not under job management control.
- Bulk - a number of records bundled in a single file to enhance throughput when processing records in asynchronous workflows.
- Bulkbuilder - An asynchronous workflow worker that accepts single records and combines them to a bulk. See Bulkbuilder documentation.
C
- Crawler - A crawler is a special worker in an asynchronous workflow that imports data from a data source (e.g. filesystem, web or database) into SMILA. It iterates over the data elements and creates records for all elements that will be further processed in the workflow. In general crawlers resp. crawl workflows are used for initial (bulk) import of data sources. (see SMILA Importing for more details)
D
- Data Object - The smallest unit of data handled by an asychronous workflow (e.g. a record bulk).
- DeltaChecker - The DeltaChecker is a worker in an (asynchronous) import workflow that handles the delta indexing.
- Delta indexing - Delta indexing is also known as incremental or generation based indexing. It is driven by DeltaChecker worker.(see SMILA Importing for more details)
E
- Eclipse - Eclipse is an open source community, whose projects are focused on building an open development platform comprised of extensible frameworks, tools and runtimes for building, deploying and managing software across the lifecycle.
- EILF - EILF (Enterprise Information Logistics Framework) was the original proposed name of SMILA. Since this abbreviation was difficult to pronounce, it was not accepted by the community and thus changed to SMILA.
- Equinox - Equinox is a base technology from Eclipse implementing the OSGi specification. Not only delivering a high performance class loading mechanism Equinox also provides an environment for managing component dependencies.
F
- Fetcher - A fetcher is a worker in an (asynchronous) import workflow that retrieves Records containing an URL or file path, etc from a crawler and actually fetches the content (e.g. of files) from the data source ((e.g. FileFetcherWorker or WebFetcherWorker)), attaches it to records and sends them to the UpdatePusher. (see SMILA Importing for more details)
G
H
I
J
- JMX - Java Management Extension is a specification to administrating and monitoring java applications.
- Job - A Job is a description of a distinct and repeatable working process that the system should accomplish. It references and parametrizes an asynchronous workflow.
- Job Run - A Job Run is an "instance" of a Job, for example one run of an import of a data source to an index. Only one active job run can existe per job. Statistics will be accumulated for each job run. A job run is automatically stopped when SMILA shuts down.
K
L
M
- Micro bulk - a (small) bundle of records in one single file which can be pushed into the system using the Bulkbuilder. The micro bulk in itself is not JSON, but a file where each line must consist of a single JSON representation of a record. E.g.:
{"_recordid": "id1", "attribute1": "attribute1", ...} {"_recordid": "id2", "attribute1": "attribute2", ...} {"_recordid": "id3", "attribute1": "attribute3", ...}
N
O
- ODE - Apache ODE (Orchestration Director Engine) executes business processes following the BPEL/WS-BPEL standard. It talks to web services, sending and receiving messages, handling data manipulation and error recovery as described by your process definition. It supports both long and short living process executions to orchestrate all the services that are part of your application.
- OSGi - The OSGi specification is about managing a component based software system. It defines an in-VM Service Oriented Architecture (SOA) for networked systems. An OSGi Service Platform provides a standardized, component-oriented computing environment for cooperating networked services. This architecture significantly reduces the overall complexity of building, maintaining, and deploying applications.
P
- Pipelet - A pipelet is a reusable component (POJO) in a BPEL workflow used to process data contained in records. See Pipelets for details.
- Pipeline - A pipeline is the definition of a BPEL process (or workflow) that orchestrates pipelets and other BPEL services (e.g. web services).
Q
R
- Record - A record is a sole element in SMILA that contains data to process (e.g. content and metadata of a document). The record consists of metadata elements, see SMILA/Documentation/Data_Model_and_Serialization_Formats.
- Record Bulk - a Data Object containing a sequence of Records
S
- Slot - An (input/output) slot is a description for the input/output behaviour of a worker. In a concrete asynchronous workflow slots are assigned to buckets
- SNMP - Simple Network Management Protocol is a network protocol which controls the communication between supervised devices and the monitoring application (e.g. JMX).
T
- Task - Description of a single unit of work to be processed by a Worker. A task can contain worker specific properties.
- Tika - The Apache Tika™ toolkit detects and extracts metadata and structured text content from various documents using existing parser libraries
U
- UpdatePusher - The UpdatePusher is a worker in an (asynchronous) import workflow that pushes the crawled records to the BulkBuilder of a running import job.
V
W
- Worker - Single processing component in an asychnronous workflow. Pulls tasks to process. Defined in a worker description.
- Worker Description - Description of a worker, e.g. its input/output slots.
- Workflow (asynchronous) - Describes an asynchronously processed workflow by specifying a sequence of workers and associating their input/output slots to buckets.
- Workflow (synchronous/BPEL) - see pipeline
- Workflow run - Single traversal of a workflow.
- WSDL - WSDL is an XML format for describing network services as a set of endpoints operating on messages containing either document-oriented or procedure-oriented information. The operations and messages are described abstractly, and then bound to a concrete network protocol and message format to define an endpoint. Related concrete endpoints are combined into abstract endpoints (services). WSDL is extensible to allow description of endpoints and their messages regardless of what message formats or network protocols are used to communicate.