Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/Processing/JSON REST API for BPEL pipelines"

(Pipeline Overview)
 
(31 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{note| Available since SMILA 0.9!}}
 
 
 
== JSON REST API for SMILA BPEL pipelines  ==
 
== JSON REST API for SMILA BPEL pipelines  ==
  
SMILA now has an HTTP REST API that allows managing and invoking BPEL processing workflows.  
+
SMILA now has an HTTP REST API that allows managing and invoking BPEL processing workflows.
  
 
=== Reference  ===
 
=== Reference  ===
Line 9: Line 7:
 
Note: The trailing slash in URLs is optional.  
 
Note: The trailing slash in URLs is optional.  
  
==== Pipeline Overview or Add/Update Pipeline ====
+
==== Get Pipeline Overview or Add/Update a Pipeline ====
  
GET: Returns a listing of all currently deployed BPEL pipelines including URLs to access their definition.  
+
GET: Returns a list of all deployed BPEL pipelines including URLs to access their definition.  
  
POST: Add/Update BPEL pipeline.
+
POST: Adds or updates a BPEL pipeline. Returns an object containing the timestamp of the creation/modification and a URL to the pipeline definition.  
  
 
'''Supported operations:'''  
 
'''Supported operations:'''  
  
 
*GET: Gets a list of all available BPEL pipelines.
 
*GET: Gets a list of all available BPEL pipelines.
*POST: Adds or updates (if already existing) BPEL pipeline. If the respective pipeline is in use, the update process will take a little bit longer.
+
*POST: Adds or updates (if already existing) a BPEL pipeline. The request JSON object consists of a "name" and a "definition" field. The latter contains the pipeline description in BPEL format. If the respective pipeline is in use, the update process will take a little longer. Also, new invocations of this pipeline are blocked until the update is finished (about 100ms usually).  
  
 
'''Usage:'''  
 
'''Usage:'''  
Line 25: Line 23:
 
*Allowed methods:  
 
*Allowed methods:  
 
**GET (no further URL parameters and no request body allowed)
 
**GET (no further URL parameters and no request body allowed)
**POST (request body is mandatory)
+
**POST (request body with a "name" and "definition" field is mandatory)
 
*Response status codes:  
 
*Response status codes:  
 
**200 OK: Upon successful execution of GET.
 
**200 OK: Upon successful execution of GET.
 
**201 CREATED: Upon successful creation/update of the pipeline.  
 
**201 CREATED: Upon successful creation/update of the pipeline.  
**400 BAD REQUEST: If you post with an empty body, try to update a system pipeline (predefined pipeline) or your submitted request body has syntax errors.
+
**400 BAD REQUEST: If you post with an empty body, if you try to update a predefined pipeline, if the request body has syntax errors or the name is invalid.
**500 INTERNAL SERVER ERROR: Any other error
+
**500 INTERNAL SERVER ERROR: Any other error.
  
==== Get or Delete Pipeline Definition  ====
+
==== Get or Delete a Pipeline Definition  ====
  
Get: Returns a JSON object containing the definition of the requested pipeline. The object consists of the name of the pipeline and the BPEL XML code as a single string value.
+
GET: Returns a JSON object containing the definition of the requested pipeline. The object consists of the name of the pipeline and the BPEL XML definition as a single string value. If the pipeline is predefined in the system configuration, the object will also contain a <tt>"readOnly": true</tt> flag. Otherwise, it will contain the <tt>timestamp</tt> of the latest add operation (see above) that created this version of the pipeline.
  
Delete: Deletes the requested BPEL pipeline.  
+
DELETE: Deletes the requested BPEL pipeline.  
  
 
'''Supported operations:'''  
 
'''Supported operations:'''  
  
*GET: Gets a BPEL pipeline definition.
+
*GET: Get a BPEL pipeline definition.
 
*DELETE: Delete the specified BPEL pipeline.
 
*DELETE: Delete the specified BPEL pipeline.
  
Line 51: Line 49:
 
*Response status codes:  
 
*Response status codes:  
 
**200 OK: Upon successful execution  for GET and DELETE. If you try to delete a pipeline which does not exist, you will get 200 OK, too.
 
**200 OK: Upon successful execution  for GET and DELETE. If you try to delete a pipeline which does not exist, you will get 200 OK, too.
**404 NOT FOUND: If the specified BPEL pipeline does not exist
+
**404 NOT FOUND: If the specified BPEL pipeline does not exist.
**500 INTERNAL SERVER ERROR: Any other error
+
**500 INTERNAL SERVER ERROR: Any other error.
  
==== Invoke Pipeline  ====
+
==== Invoke a Pipeline  ====
  
Process a record with the selected pipeline. Returns the result record as a JSON object.  
+
Process a record with the selected pipeline. Returns the result record as a JSON object. Record attachments are supported by using Multipart POST requests, see [[SMILA/Documentation/JettyHttpServer#Attachments]] for details and code example.  
  
 
'''Supported operations:'''  
 
'''Supported operations:'''  
Line 77: Line 75:
 
===== Current Limitations  =====
 
===== Current Limitations  =====
  
*Exactly one non-empty input record must be sent, attachments cannot be added.
+
* Only the first result record is returned, no attachments will be contained.
*Only the first result record is returned, attachments are ignored.  
+
* The request is executed using a pure transient blackboard, so the records and attachments are not persisted to <tt>RecordStore</tt> and <tt>BinaryStorage</tt>, respectively.
*The request is executed using a transient blackboard, so the records are not persisted to <tt>RecordStore</tt>.
+
* The name of the pipeline must be a valid name (see [[SMILA/Documentation/BPEL Workflow Processor#Pipeline_definition_using_BPEL]]
  
 
=== Example Walkthrough  ===
 
=== Example Walkthrough  ===
  
 
{{note| Some Tool recommendations
 
{{note| Some Tool recommendations
You can use tools like [http://curl.haxx.se/ CURL] or [https://github.com/micha/resty Resty] to invoke HTTP APIs from the command line. More convenient are browser extensions like [https://addons.mozilla.org/de/firefox/addon/restclient/ RESTClient] or [https://addons.mozilla.org/de/firefox/addon/poster/ Poster]for Firefox, [https://chrome.google.com/webstore/detail/baedhhmoaooldchehjhlpppaieoglhml cREST] for Chrome or [https://addons.opera.com/addons/extensions/details/simple-rest-client/1.3/ Simple REST Client] for Opera. And, of course, there are Plugins for Eclipse like [http://marketplace.eclipse.org/content/rest-client restclient-tool].
+
See [[SMILA/Documentation/Using_The_ReST_API]] for recommendations on how to use the ReST API manually.
  
 
For sake of simplicity, the following examples were produced with Resty.}}  
 
For sake of simplicity, the following examples were produced with Resty.}}  
  
 
Preparation: Start SMILA and index some documents. E.g. follow the instructions on [[SMILA/Documentation for 5 Minutes to Success]].  
 
Preparation: Start SMILA and index some documents. E.g. follow the instructions on [[SMILA/Documentation for 5 Minutes to Success]].  
 +
 +
==== Accessing Pipelines ====
  
 
Let's start with the overview:  
 
Let's start with the overview:  
Line 101: Line 101:
 
     "url" : "http://localhost:8080/smila/pipeline/AddFeedPipeline/"
 
     "url" : "http://localhost:8080/smila/pipeline/AddFeedPipeline/"
 
   }, {
 
   }, {
     "name" : "XmlDumpSplitterPipeline",
+
     "name" : "XmlSplitAndAddPipeline",
     "url" : "http://localhost:8080/smila/pipeline/XmlDumpSplitterPipeline/"
+
     "url" : "http://localhost:8080/smila/pipeline/XmlSplitAndAddPipeline/"
 
   }, {
 
   }, {
 
     "name" : "DeletePipeline",
 
     "name" : "DeletePipeline",
Line 110: Line 110:
 
     "url" : "http://localhost:8080/smila/pipeline/SearchPipeline/"
 
     "url" : "http://localhost:8080/smila/pipeline/SearchPipeline/"
 
   }, {
 
   }, {
     "name" : "XmlDumpAddPipeline",
+
     "name" : "AdaptFileCrawlerWorkerOutput",
     "url" : "http://localhost:8080/smila/pipeline/XmlDumpAddPipeline/"
+
     "url" : "http://localhost:8080/smila/pipeline/AdaptFileCrawlerWorkerOutput/"
 +
  }, {
 +
    "name" : "AdaptWebCrawlerWorkerOutput",
 +
    "url" : "http://localhost:8080/smila/pipeline/AdaptWebCrawlerWorkerOutput/"
 
   }, {
 
   }, {
 
     "name" : "AddPipeline",
 
     "name" : "AddPipeline",
Line 123: Line 126:
 
<source lang="javascript">
 
<source lang="javascript">
 
> GET /smila/pipeline/SearchPipeline/
 
> GET /smila/pipeline/SearchPipeline/
{
+
{"name":"SearchPipeline","readOnly":true,"definition":"<?xml version=\"1.0\" encoding=\"utf-
  "name" : "SearchPipeline",
+
8\" ?>\r\n<!--\r\n  * Copyright (c) 2009 empolis GmbH and brox IT Solutions GmbH.\r\n  * All
  "definition" : "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\r\n<!--\r\n  * Copyright (c) 2009 empolis GmbH and brox IT
+
rights reserved. This program and the accompanying materials\r\n  * are made available unde
Solutions GmbH.\r\n  * All rights reserved. This program and the accompanying materials\r\n  * are made available under
+
r the terms of the Eclipse Public License v1.0\r\n  * which accompanies this distribution, a
the terms of the Eclipse Public License v1.0\r\n  * which accompanies this distribution, and is available at\r\n  *  
+
nd is available at\r\n  * http://www.eclipse.org/legal/epl-v10.html\r\n  *\r\n  * Contributo
http://www.eclipse.org/legal/epl-v10.html\r\n  *\r\n  * Contributors:\r\n  * Juergen Schumacher (empolis GmbH) -  
+
rs:\r\n  * Juergen Schumacher (empolis GmbH) - initial design\r\n-->\r\n<process name=\"Sear
initial design\r\n-->\r\n<process name=\"SearchPipeline\" targetNamespace=\"http://www.eclipse.org/smila/processor\"\r\n   
+
chPipeline\" targetNamespace=\"http://www.eclipse.org/smila/processor\"\r\n  xmlns=\"http://
xmlns=\"http://docs.oasis-open.org/wsbpel/2.0/process/executable\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"\r\n   
+
docs.oasis-open.org/wsbpel/2.0/process/executable\" xmlns:xsd=\"http://www.w3.org/2001/XMLSc
xmlns:proc=\"http://www.eclipse.org/smila/processor\" xmlns:rec=\"http://www.eclipse.org/smila/record\"\r\n   
+
hema\"\r\n  xmlns:proc=\"http://www.eclipse.org/smila/processor\" xmlns:rec=\"http://www.ecl
xmlns:bpel=\"http://docs.oasis-open.org/wsbpel/2.0/process/executable\">\r\n\r\n  <import location=\"processor.wsdl\"  
+
ipse.org/smila/record\"\r\n  xmlns:bpel=\"http://docs.oasis-open.org/wsbpel/2.0/process/exec
namespace=\"http://www.eclipse.org/smila/processor\"\r\n    importType=\"http://schemas.xmlsoap.org/wsdl/\" />\r\n\r\n
+
utable\">\r\n\r\n  <import location=\"processor.wsdl\" namespace=\"http://www.eclipse.org/sm
  <partnerLinks>\r\n    <partnerLink name=\"Pipeline\" partnerLinkType=\"proc:ProcessorPartnerLinkType\" myRole=\"service\"  
+
ila/processor\"\r\n    importType=\"http://schemas.xmlsoap.org/wsdl/\" />\r\n\r\n <partnerL
/>\r\n  </partnerLinks>\r\n\r\n  <extensions>\r\n    <extension namespace=\"http://www.eclipse.org/smila/processor\"  
+
inks>\r\n    <partnerLink name=\"Pipeline\" partnerLinkType=\"proc:ProcessorPartnerLinkType\
mustUnderstand=\"no\" />\r\n  </extensions>\r\n\r\n  <variables>\r\n    <variable name=\"request\" messageType=
+
" myRole=\"service\" />\r\n  </partnerLinks>\r\n\r\n  <extensions>\r\n    <extension namespa
\"proc:ProcessorMessage\" />\r\n  </variables>\r\n\r\n  <sequence name=\"SearchPipeline\">\r\n    <receive name=\"start\"  
+
ce=\"http://www.eclipse.org/smila/processor\" mustUnderstand=\"no\" />\r\n  </extensions>\r\
partnerLink=\"Pipeline\" portType=\"proc:ProcessorPortType\"\r\n      operation=\"process\" variable=\"request\"  
+
n\r\n  <variables>\r\n    <variable name=\"request\" messageType=\"proc:ProcessorMessage\" /
createInstance=\"yes\" />\r\n\r\n    <extensionActivity>\r\n      <proc:invokePipelet name=\"search\">\r\n         
+
>\r\n  </variables>\r\n\r\n  <sequence name=\"SearchPipeline\">\r\n    <receive name=\"start
<proc:pipelet class=\"org.eclipse.smila.lucene.pipelets.LuceneSearchPipelet\" />\r\n        <proc:variables input=\"request\"  
+
\" partnerLink=\"Pipeline\" portType=\"proc:ProcessorPortType\"\r\n      operation=\"process
/>\r\n        <proc:configuration>\r\n         <rec:Val key=\"indexname\">test_index</rec:Val>\r\n      
+
\" variable=\"request\" createInstance=\"yes\" />\r\n\r\n    <extensionActivity>\r\n      <p
</proc:configuration>\r\n      </proc:invokePipelet>\r\n    </extensionActivity>\r\n\r\n    <extensionActivity>\r\n
+
roc:invokePipelet name=\"invokeSolrSearchPipelet\">\r\n        <proc:pipelet class=
      <proc:invokePipelet name=\"highlight results\">\r\n        <proc:pipelet class=
+
\"org.eclipse.smila.solr.search.SolrSearchPipelet\" />\r\n        <proc:variables input=\"re
\"org.eclipse.smila.search.highlighting.HighlightingPipelet\" />\r\n       <proc:variables input=\"request\" />\r\n   
+
quest\" output=\"request\" />\r\n        <proc:configuration>\r\n           <rec:Val key=\"
    <proc:configuration>\r\n          <rec:Map key=\"highlightingTransformers\">\r\n           <rec:Map key=\"Content\">
+
indexname\">DefaultCore</rec:Val>\r\n           <rec:Map key=\"_solr.query\">\r\n        
\r\n             <rec:Val key=\"name\">Sentence</rec:Val>\r\n             <rec:Val key=\"MaxLength\">300</rec:Val>\r
+
      <rec:Seq key=\"highlighting\">           \r\n                   <rec:Map>\r\n      
\n              <rec:Val key=\"MarkupPrefix\"><b></rec:Val>\r\n             <rec:Val key=\"MarkupSuffix\"></b>
+
              <rec:Val key=\"attribute\">global.solr.params</rec:Val>\r\n                
</rec:Val>\r\n             <rec:Val key=\"MaxHLElements\">999</rec:Val>\r\n              <rec:Val key=\"MaxSucceedingCharacters
+
    <rec:Val key=\"hl\" type=\"boolean\">true</rec:Val>\r\n                     <rec:Val ke
\">30</rec:Val>\r\n             <rec:Val key=\"SucceedingCharacters\">...</rec:Val>\r\n              <rec:Val
+
y=\"hl.fl\">Content</rec:Val>\r\n                     <rec:Val key=\"hl.simple.pre\">&lt;b&
key=\"SortAlgorithm\">Occurrence</rec:Val>\r\n             <rec:Val key=\"TextHandling\">ReturnSnipplet</rec:Val>\r\n  
+
gt;</rec:Val>\r\n                     <rec:Val key=\"hl.simple.post\">&lt;/b&gt;</rec:Val>  
          </rec:Map>\r\n         </rec:Map>\r\n        </proc:configuration>\r\n      </proc:invokePipelet>\r\n     
+
            \r\n                   </rec:Map>\r\n               </rec:Seq>              
</extensionActivity>\r\n\r\n    <reply name=\"end\" partnerLink=\"Pipeline\" portType=\"proc:ProcessorPortType\"  
+
 
operation=\"process\"\r\n      variable=\"request\" />\r\n    <exit />\r\n  </sequence>\r\n</process>\r\n"
+
    \r\n           </rec:Map>               \r\n        </proc:configuration>\r\n      </p
}
+
roc:invokePipelet>\r\n    </extensionActivity>\r\n   \r\n    <reply name=\"end\" partnerLin
 +
k=\"Pipeline\" portType=\"proc:ProcessorPortType\" operation=\"process\"\r\n      variable=\
 +
"request\" />\r\n  </sequence>\r\n</process>\r\n"}
 
</source>  
 
</source>  
  
Whoops, what's this? It's a bit awkward to read because newline and double-quote characters are printed in their JSON-escaped form. In a browser with a [[https://addons.mozilla.org/en-US/firefox/addon/jsonview/ JSONView]] extension installed it looks quite readable, e.g. in Firefox:  
+
Whoops, what's this? It's a bit awkward to read because newline and double-quote characters are printed in their JSON-escaped form. In a browser with a "JSONView" extension installed it looks quite readable, e.g. in Chrome:  
 +
 
 +
[[Image:SMILA-bpel-pipeline-in-jsonview-browser.png]]
  
[[Image:SMILA-bpel-pipeline-in-jsonview-firefox.png]]
+
That's better. We see that the object has a <tt>readOnly</tt> flag set to true, because it is one of the predefined pipelines in the system configuration. If it was a custom pipeline defined via the API, there would be a <tt>timestamp</tt> attribute at the end of the object.
  
That's better. Now we try to execute this pipeline:  
+
Now we try to execute this pipeline:  
  
 
<source lang="javascript">
 
<source lang="javascript">
Line 222: Line 229:
 
</source>  
 
</source>  
  
A successful search ... but we probably want to see a bit more information. So we add values for parameter "resultAttributes":  
+
A successful search ... but we probably want to see a bit more information. So we add values for the parameter <tt>resultAttributes</tt>:  
  
 
<source lang="javascript">
 
<source lang="javascript">
Line 293: Line 300:
 
<tt>http://localhost:8080/smila/pipeline/SearchPipeline/process?query=SMILA&amp;QueryAttribute=Content</tt>  
 
<tt>http://localhost:8080/smila/pipeline/SearchPipeline/process?query=SMILA&amp;QueryAttribute=Content</tt>  
  
However, this gets inconvenient if you want to add lots of parameters and attributes.
+
However, this gets inconvenient when you want to add lots of parameters and attributes.
 +
 
 +
==== Defining Pipelines ====
 +
 
 +
To define a new pipeline (or update it) you POST a JSON containing the pipeline name and BPEL definition to <code>/smila/pipeline</code>. E.g. copy the output of the <code>GET /smila/pipeline/SearchPipeline</code> command and rename the pipeline to "SearchPipeline2" like this:
 +
 
 +
<source lang="javascript">
 +
> POST /smila/pipeline/ '{
 +
  "name" : "SearchPipeline2",
 +
  "definition" : "<?xml version=\"1.0\" ?>\r\n<process name=\"SearchPipeline2\"  ..."
 +
}'
 +
{
 +
  "name" : "SearchPipeline2",
 +
  "timestamp" : "2011-08-26T13:54:34.451+0200",
 +
  "url" : "http://localhost:8080/smila/pipeline/SearchPipeline2/"
 +
}
 +
</source>
 +
 
 +
If you want to push an own BPEL definition, take care of escaping linefeed (\r), newline (\n) and quotes (\") characters, or the JSON code will not be valid.
 +
 
 +
The response contains the name, a creation timestamp and an URL to read the pipeline definition again. If you use this, you will see that the response will also contain the timestamp, but no readOnly flag:
 +
 
 +
<source lang="javascript">
 +
> GET /smila/pipeline/SearchPipeline2/
 +
{
 +
  "name" : "SearchPipeline2",
 +
  "definition" : "<?xml version=\"1.0\" ?>\r\n<process name=\"SearchPipeline2\" ...",
 +
  "timestamp" : "2011-08-26T13:54:34.451+0200"
 +
}
 +
</source>
 +
 
 +
Updating the workflow would work just the same. The timestamp can be used in modelling tools to ensure that different users do not overwrite changes made by another user.
 +
 
 +
The new pipeline should now also appears in the list of pipelines:
 +
 
 +
<source lang="javascript">
 +
> GET /smila/pipeline/
 +
{
 +
  "pipelines" : [ {
 +
  ...,
 +
  {
 +
    "name" : "SearchPipeline",
 +
    "url" : "http://localhost:8080/smila/pipeline/SearchPipeline/"
 +
  }, {
 +
    "name" : "SearchPipeline2",
 +
    "url" : "http://localhost:8080/smila/pipeline/SearchPipeline2/"
 +
  },
 +
  ...]
 +
}
 +
 
 +
</source>

Latest revision as of 09:56, 26 January 2012

JSON REST API for SMILA BPEL pipelines

SMILA now has an HTTP REST API that allows managing and invoking BPEL processing workflows.

Reference

Note: The trailing slash in URLs is optional.

Get Pipeline Overview or Add/Update a Pipeline

GET: Returns a list of all deployed BPEL pipelines including URLs to access their definition.

POST: Adds or updates a BPEL pipeline. Returns an object containing the timestamp of the creation/modification and a URL to the pipeline definition.

Supported operations:

  • GET: Gets a list of all available BPEL pipelines.
  • POST: Adds or updates (if already existing) a BPEL pipeline. The request JSON object consists of a "name" and a "definition" field. The latter contains the pipeline description in BPEL format. If the respective pipeline is in use, the update process will take a little longer. Also, new invocations of this pipeline are blocked until the update is finished (about 100ms usually).

Usage:

  • URL: http://<hostname>:8080/smila/pipeline/
  • Allowed methods:
    • GET (no further URL parameters and no request body allowed)
    • POST (request body with a "name" and "definition" field is mandatory)
  • Response status codes:
    • 200 OK: Upon successful execution of GET.
    • 201 CREATED: Upon successful creation/update of the pipeline.
    • 400 BAD REQUEST: If you post with an empty body, if you try to update a predefined pipeline, if the request body has syntax errors or the name is invalid.
    • 500 INTERNAL SERVER ERROR: Any other error.

Get or Delete a Pipeline Definition

GET: Returns a JSON object containing the definition of the requested pipeline. The object consists of the name of the pipeline and the BPEL XML definition as a single string value. If the pipeline is predefined in the system configuration, the object will also contain a "readOnly": true flag. Otherwise, it will contain the timestamp of the latest add operation (see above) that created this version of the pipeline.

DELETE: Deletes the requested BPEL pipeline.

Supported operations:

  • GET: Get a BPEL pipeline definition.
  • DELETE: Delete the specified BPEL pipeline.

Usage:

  • URL: http://<hostname>:8080/smila/pipeline/<workflow-name>/
  • Allowed methods:
    • GET (no further URL parameters and no request body allowed)
    • DELETE (no further URL parameters and no request body allowed)
  • Response status codes:
    • 200 OK: Upon successful execution for GET and DELETE. If you try to delete a pipeline which does not exist, you will get 200 OK, too.
    • 404 NOT FOUND: If the specified BPEL pipeline does not exist.
    • 500 INTERNAL SERVER ERROR: Any other error.

Invoke a Pipeline

Process a record with the selected pipeline. Returns the result record as a JSON object. Record attachments are supported by using Multipart POST requests, see SMILA/Documentation/JettyHttpServer#Attachments for details and code example.

Supported operations:

  • GET: Invokes a pipeline where the record to process is specified using URL parameters.
  • POST: Invokes a pipeline where the record to process is contained in the request body as JSON (recommended).

Usage:

  • URL: http://<hostname>:8080/smila/pipeline/<workflow-name>/process/
  • Allowed methods:
    • POST
    • GET
  • Response status codes:
    • 200 OK: Upon successful execution
    • 400 BAD REQUEST: In case of empty input records or invalid JSON.
    • 405 METHOD NOT ALLOWED: In case of an invalid HTTP method (i.e. other than GET or POST).
    • 500 INTERNAL SERVER ERROR: Other errors, e.g. error during processing.
Current Limitations

Example Walkthrough

Note.png
Some Tool recommendations

See SMILA/Documentation/Using_The_ReST_API for recommendations on how to use the ReST API manually.

For sake of simplicity, the following examples were produced with Resty.


Preparation: Start SMILA and index some documents. E.g. follow the instructions on SMILA/Documentation for 5 Minutes to Success.

Accessing Pipelines

Let's start with the overview:

> resty http://localhost:8080
http://localhost:8080*
> GET /smila/pipeline/
{
  "pipelines" : [ {
    "name" : "AddFeedPipeline",
    "url" : "http://localhost:8080/smila/pipeline/AddFeedPipeline/"
  }, {
    "name" : "XmlSplitAndAddPipeline",
    "url" : "http://localhost:8080/smila/pipeline/XmlSplitAndAddPipeline/"
  }, {
    "name" : "DeletePipeline",
    "url" : "http://localhost:8080/smila/pipeline/DeletePipeline/"
  }, {
    "name" : "SearchPipeline",
    "url" : "http://localhost:8080/smila/pipeline/SearchPipeline/"
  }, {
    "name" : "AdaptFileCrawlerWorkerOutput",
    "url" : "http://localhost:8080/smila/pipeline/AdaptFileCrawlerWorkerOutput/"
  }, {
    "name" : "AdaptWebCrawlerWorkerOutput",
    "url" : "http://localhost:8080/smila/pipeline/AdaptWebCrawlerWorkerOutput/"
  }, {
    "name" : "AddPipeline",
    "url" : "http://localhost:8080/smila/pipeline/AddPipeline/"
  } ]
}

So we have seven pipelines deployed. Fine. Let's have a look at their definitions:

> GET /smila/pipeline/SearchPipeline/
{"name":"SearchPipeline","readOnly":true,"definition":"<?xml version=\"1.0\" encoding=\"utf-
8\" ?>\r\n<!--\r\n  * Copyright (c) 2009 empolis GmbH and brox IT Solutions GmbH.\r\n  * All
 rights reserved. This program and the accompanying materials\r\n  * are made available unde
r the terms of the Eclipse Public License v1.0\r\n  * which accompanies this distribution, a
nd is available at\r\n  * http://www.eclipse.org/legal/epl-v10.html\r\n  *\r\n  * Contributo
rs:\r\n  * Juergen Schumacher (empolis GmbH) - initial design\r\n-->\r\n<process name=\"Sear
chPipeline\" targetNamespace=\"http://www.eclipse.org/smila/processor\"\r\n  xmlns=\"http://
docs.oasis-open.org/wsbpel/2.0/process/executable\" xmlns:xsd=\"http://www.w3.org/2001/XMLSc
hema\"\r\n  xmlns:proc=\"http://www.eclipse.org/smila/processor\" xmlns:rec=\"http://www.ecl
ipse.org/smila/record\"\r\n  xmlns:bpel=\"http://docs.oasis-open.org/wsbpel/2.0/process/exec
utable\">\r\n\r\n  <import location=\"processor.wsdl\" namespace=\"http://www.eclipse.org/sm
ila/processor\"\r\n    importType=\"http://schemas.xmlsoap.org/wsdl/\" />\r\n\r\n  <partnerL
inks>\r\n    <partnerLink name=\"Pipeline\" partnerLinkType=\"proc:ProcessorPartnerLinkType\
" myRole=\"service\" />\r\n  </partnerLinks>\r\n\r\n  <extensions>\r\n    <extension namespa
ce=\"http://www.eclipse.org/smila/processor\" mustUnderstand=\"no\" />\r\n  </extensions>\r\
n\r\n  <variables>\r\n    <variable name=\"request\" messageType=\"proc:ProcessorMessage\" /
>\r\n  </variables>\r\n\r\n  <sequence name=\"SearchPipeline\">\r\n    <receive name=\"start
\" partnerLink=\"Pipeline\" portType=\"proc:ProcessorPortType\"\r\n      operation=\"process
\" variable=\"request\" createInstance=\"yes\" />\r\n\r\n    <extensionActivity>\r\n      <p
roc:invokePipelet name=\"invokeSolrSearchPipelet\">\r\n        <proc:pipelet class=
\"org.eclipse.smila.solr.search.SolrSearchPipelet\" />\r\n        <proc:variables input=\"re
quest\" output=\"request\" />\r\n        <proc:configuration>\r\n            <rec:Val key=\"
indexname\">DefaultCore</rec:Val>\r\n            <rec:Map key=\"_solr.query\">\r\n          
      <rec:Seq key=\"highlighting\">            \r\n                    <rec:Map>\r\n       
               <rec:Val key=\"attribute\">global.solr.params</rec:Val>\r\n                  
    <rec:Val key=\"hl\" type=\"boolean\">true</rec:Val>\r\n                      <rec:Val ke
y=\"hl.fl\">Content</rec:Val>\r\n                      <rec:Val key=\"hl.simple.pre\">&lt;b&
gt;</rec:Val>\r\n                      <rec:Val key=\"hl.simple.post\">&lt;/b&gt;</rec:Val> 
            \r\n                    </rec:Map>\r\n                </rec:Seq>                
 
    \r\n            </rec:Map>                \r\n        </proc:configuration>\r\n      </p
roc:invokePipelet>\r\n    </extensionActivity>\r\n    \r\n    <reply name=\"end\" partnerLin
k=\"Pipeline\" portType=\"proc:ProcessorPortType\" operation=\"process\"\r\n      variable=\
"request\" />\r\n  </sequence>\r\n</process>\r\n"}

Whoops, what's this? It's a bit awkward to read because newline and double-quote characters are printed in their JSON-escaped form. In a browser with a "JSONView" extension installed it looks quite readable, e.g. in Chrome:

SMILA-bpel-pipeline-in-jsonview-browser.png

That's better. We see that the object has a readOnly flag set to true, because it is one of the predefined pipelines in the system configuration. If it was a custom pipeline defined via the API, there would be a timestamp attribute at the end of the object.

Now we try to execute this pipeline:

> GET /smila/pipeline/SearchPipeline/process/
{
  "message" : "Cannot process an empty record."
}
> POST /smila/pipeline/SearchPipeline/process/
{
  "message" : "Cannot process an empty record."
}
So this did not work, as expected. In the response headers you can see the error code returned:
> POST /smila/pipeline/SearchPipeline/process/ -v
...
< HTTP/1.1 400 Bad Request
...
{
  "message" : "Cannot process an empty record."
}

So we add a query attribute:

> POST /smila/pipeline/SearchPipeline/process/ '{ "query": "SMILA" }'
...
< HTTP/1.1 500 Server Error
...
{
  "message" : "Error processing BPEL workflow SearchPipeline: Invocation of pipeline element SearchPipeline/search failed: Error processing message SearchPipeline-7afe423a-749c-4492-aa66-38ce37dba672\ncaused by: Invocation of pipeline element SearchPipeline/search failed: Error processing message SearchPipeline-7afe423a-749c-4492-aa66-38ce37dba672\ncaused by: no single value for required parameter QueryAttribute"
}

This time the LuceneSearchPipelet complains about a missing parameter. So let's add it:

> POST /smila/pipeline/SearchPipeline/process/ '{ 
    "query": "SMILA", 
    "QueryAttribute": "Content" }' 
{
  "query" : "SMILA",
  "QueryAttribute" : "Content",
  "_recordid" : "SearchPipeline-5c2d3f3f-1e56-4362-aa4c-74aa5fa9d6e8",
  "count" : 58,
  "indexSize" : 115,
  "records" : [ {
    "_recordid" : "feeds:<Uri=tag:search.twitter.com,2005:69739397733560320>",
    "_source" : "feeds",
    "_weight" : 0.84
  }, {
    "_recordid" : "feeds:<Uri=tag:search.twitter.com,2005:69960011966713856>",
    "_source" : "feeds",
    "_weight" : 0.73
  }, 
  ...
  ] }
}

A successful search ... but we probably want to see a bit more information. So we add values for the parameter resultAttributes:

> POST /smila/pipeline/SearchPipeline/process/ '{ 
  "query": "SMILA", 
  "QueryAttribute": "Content", 
  "resultAttributes": [ "Title", "Author", "LastModifiedDate" ] }' 
{
  "query" : "SMILA",
  "QueryAttribute" : "Content",
  "resultAttributes" : [ "Title", "Author", "LastModifiedDate" ],
  "_recordid" : "SearchPipeline-2201c83a-a9b6-4cfe-a62e-6ec8ffe113ca",
  "count" : 58,
  "indexSize" : 115,
  "records" : [ {
    "_recordid" : "feeds:<Uri=tag:search.twitter.com,2005:69739397733560320>",
    "_source" : "feeds",
    "_weight" : 0.84,
    "Title" : "@AbrarAlAdwani smila 3alech. Latwaswiseen",
    "Author" : "dee_the_bee (Dalalee Boland)",
    "LastModifiedDate" : "2011-05-15T14:22:22.000+0200"
  }, {
    "_recordid" : "feeds:<Uri=tag:search.twitter.com,2005:69960011966713856>",
    "_source" : "feeds",
    "_weight" : 0.73,
    "Title" : "Finally found something to smila bout :)",
    "Author" : "MIGUELALMENDRAL (Miguel FG Almendral)",
    "LastModifiedDate" : "2011-05-16T04:59:01.000+0200"
  },
  ...
  ] }
}

Awesome! In the same way you can add more parameters as you need:

> POST /smila/pipeline/SearchPipeline/process/ '{ 
    "query": "SMILA", 
    "QueryAttribute": "Content", 
    "resultAttributes": [ "Title", "Author", "LastModifiedDate" ], 
    "maxcount": 1, "offset": 42, "highlight": "Content" }' 
{
  "query" : "SMILA",
  "QueryAttribute" : "Content",
  "resultAttributes" : [ "Title", "Author", "LastModifiedDate" ],
  "maxcount" : 1,
  "offset" : 42,
  "highlight" : "Content",
  "_recordid" : "SearchPipeline-f2a53434-36b1-4e25-a716-f8c02fba5ecd",
  "count" : 58,
  "indexSize" : 115,
  "records" : [ {
    "_recordid" : "feeds:<Uri=http://www.eclipse.org/forums/index.php/mv/msg/206311/660699/#msg_660699>",
    "_source" : "feeds",
    "_weight" : 0.19,
    "Title" : "Re: New SMILA tryout",
    "Author" : "Andreas Weber",
    "_highlight" : {
      "Content" : {
        "text" : "<br />\n&#62;<br />\n&#62; I've done all standard steps as we do when creating a new<br />\n&#62; <b>SMILA</b>-workspace (and as described...-workspace (and as described on the <b>SMILA</b> website - dont know the<br... website - dont know the<br />\n&#62; exact name right now):<br />\n&#62;<br />\n&#62;<br />\n&#62; check out from eclipse SVN<br />\n&#62; add a new Target Platform with path to the <b>SMILA</b> bundles and to eclipse<br..."
      }
    }
  } ]
}

You can also use GET and URL parameters to invoke the pipeline, just enter something like this in the address line of your favorite browser:

http://localhost:8080/smila/pipeline/SearchPipeline/process?query=SMILA&QueryAttribute=Content

However, this gets inconvenient when you want to add lots of parameters and attributes.

Defining Pipelines

To define a new pipeline (or update it) you POST a JSON containing the pipeline name and BPEL definition to /smila/pipeline. E.g. copy the output of the GET /smila/pipeline/SearchPipeline command and rename the pipeline to "SearchPipeline2" like this:

> POST /smila/pipeline/ '{
  "name" : "SearchPipeline2",
  "definition" : "<?xml version=\"1.0\" ?>\r\n<process name=\"SearchPipeline2\"  ..."
}'
{
  "name" : "SearchPipeline2",
  "timestamp" : "2011-08-26T13:54:34.451+0200",
  "url" : "http://localhost:8080/smila/pipeline/SearchPipeline2/"
}

If you want to push an own BPEL definition, take care of escaping linefeed (\r), newline (\n) and quotes (\") characters, or the JSON code will not be valid.

The response contains the name, a creation timestamp and an URL to read the pipeline definition again. If you use this, you will see that the response will also contain the timestamp, but no readOnly flag:

> GET /smila/pipeline/SearchPipeline2/
{
  "name" : "SearchPipeline2",
  "definition" : "<?xml version=\"1.0\" ?>\r\n<process name=\"SearchPipeline2\" ...",
  "timestamp" : "2011-08-26T13:54:34.451+0200"
}

Updating the workflow would work just the same. The timestamp can be used in modelling tools to ensure that different users do not overwrite changes made by another user.

The new pipeline should now also appears in the list of pipelines:

> GET /smila/pipeline/
{
  "pipelines" : [ {
  ...,
  {
    "name" : "SearchPipeline",
    "url" : "http://localhost:8080/smila/pipeline/SearchPipeline/"
  }, {
    "name" : "SearchPipeline2",
    "url" : "http://localhost:8080/smila/pipeline/SearchPipeline2/"
  },
  ...]
}

Back to the top