SMILA/Documentation/Processing/JSON REST API for BPEL pipelines

Available since SMILA 0.9!

JSON REST API for SMILA BPEL pipelines

SMILA now has an HTTP REST API that allows managing and invoking BPEL processing workflows.

Reference

Note: The trailing slash in URLs is optional.

Pipeline Overview or Add/Update Pipeline

GET: Returns a listing of all currently deployed BPEL pipelines including URLs to access their definition.

POST: Add/Update BPEL pipeline. Returns an object containing a timestamp for the update and the URL to get the pipeline definition.

Supported operations:

• GET: Gets a list of all available BPEL pipelines.
• POST: Adds or updates (if already existing) BPEL pipeline. The request JSON object consists of a "name" string and a "definition" string where "definition" is the BPEL code. If the respective pipeline is in use, the update process will take a little bit longer, also new invocations of this pipeline are blocked until the update is finished (about 100ms usually). In case of success the returned object will look like this:

{
    "name": "SearchPipeline2",
    "timestamp": "2011-08-25T15:58:10.722+0200",
    "url": "http://localhost:8080/smila/pipeline/SearchPipeline2/"
}

Usage:

• URL: http://<hostname>:8080/smila/pipeline/
• Allowed methods:
  • GET (no further URL parameters and no request body allowed)
  • POST (request body is mandatory)
• Response status codes:
  • 200 OK: Upon successful execution of GET.
  • 201 CREATED: Upon successful creation/update of the pipeline.
  • 400 BAD REQUEST: If you post with an empty body, try to update a system pipeline (predefined pipeline) or your submitted request body has syntax errors.
  • 500 INTERNAL SERVER ERROR: Any other error

Get or Delete Pipeline Definition

Get: Returns a JSON object containing the definition of the requested pipeline. The object consists of the name of the pipeline and the BPEL XML code as a single string value. If the pipeline is predefined in the system configuration, the object will also contain a "readOnly": true flag. Otherwise it will contain the timestamp of the Add operation result (see above) that created this version of the pipeline.

Delete: Deletes the requested BPEL pipeline.

Supported operations:

• GET: Gets a BPEL pipeline definition.
• DELETE: Delete the specified BPEL pipeline.

Usage:

• URL: http://<hostname>:8080/smila/pipeline/<workflow-name>/
• Allowed methods:
  • GET (no further URL parameters and no request body allowed)
  • DELETE (no further URL parameters and no request body allowed)
• Response status codes:
  • 200 OK: Upon successful execution for GET and DELETE. If you try to delete a pipeline which does not exist, you will get 200 OK, too.
  • 404 NOT FOUND: If the specified BPEL pipeline does not exist
  • 500 INTERNAL SERVER ERROR: Any other error

Invoke Pipeline

Process a record with the selected pipeline. Returns the result record as a JSON object.

Supported operations:

• GET: Invokes a pipeline where the record to process is specified using URL parameters.
• POST: Invokes a pipeline where the record to process is contained in the request body as JSON (recommended).

Usage:

• URL: http://<hostname>:8080/smila/pipeline/<workflow-name>/process/
• Allowed methods:
  • POST
  • GET
• Response status codes:
  • 200 OK: Upon successful execution
  • 400 BAD REQUEST: In case of empty input records or invalid JSON.
  • 405 METHOD NOT ALLOWED: In case of an invalid HTTP method (i.e. other than GET or POST).
  • 500 INTERNAL SERVER ERROR: Other errors, e.g. error during processing.

Current Limitations

• Exactly one non-empty input record must be sent, attachments cannot be added.
• Only the first result record is returned, attachments are ignored.
• The request is executed using a transient blackboard, so the records are not persisted to RecordStore.

Example Walkthrough

Some Tool recommendations

You can use tools like CURL or Resty to invoke HTTP APIs from the command line. More convenient are browser extensions like RESTClient or Posterfor Firefox, cREST for Chrome or Simple REST Client for Opera. And, of course, there are Plugins for Eclipse like restclient-tool.

For sake of simplicity, the following examples were produced with Resty.

Preparation: Start SMILA and index some documents. E.g. follow the instructions on SMILA/Documentation for 5 Minutes to Success.

Let's start with the overview:

> resty http://localhost:8080
http://localhost:8080*
> GET /smila/pipeline/
{
  "pipelines" : [ {
    "name" : "AddFeedPipeline",
    "url" : "http://localhost:8080/smila/pipeline/AddFeedPipeline/"
  }, {
    "name" : "XmlDumpSplitterPipeline",
    "url" : "http://localhost:8080/smila/pipeline/XmlDumpSplitterPipeline/"
  }, {
    "name" : "DeletePipeline",
    "url" : "http://localhost:8080/smila/pipeline/DeletePipeline/"
  }, {
    "name" : "SearchPipeline",
    "url" : "http://localhost:8080/smila/pipeline/SearchPipeline/"
  }, {
    "name" : "XmlDumpAddPipeline",
    "url" : "http://localhost:8080/smila/pipeline/XmlDumpAddPipeline/"
  }, {
    "name" : "AddPipeline",
    "url" : "http://localhost:8080/smila/pipeline/AddPipeline/"
  } ]
}

So we have seven pipelines deployed. Fine. Let's have a look at their definitions:

> GET /smila/pipeline/SearchPipeline/
{
  "name" : "SearchPipeline",
  "readOnly": true,
  "definition" : "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\r\n<!--\r\n  * Copyright (c) 2009 empolis GmbH and brox IT
 Solutions GmbH.\r\n  * All rights reserved. This program and the accompanying materials\r\n  * are made available under 
the terms of the Eclipse Public License v1.0\r\n  * which accompanies this distribution, and is available at\r\n  * 
http://www.eclipse.org/legal/epl-v10.html\r\n  *\r\n  * Contributors:\r\n  * Juergen Schumacher (empolis GmbH) - 
initial design\r\n-->\r\n<process name=\"SearchPipeline\" targetNamespace=\"http://www.eclipse.org/smila/processor\"\r\n  
xmlns=\"http://docs.oasis-open.org/wsbpel/2.0/process/executable\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"\r\n  
xmlns:proc=\"http://www.eclipse.org/smila/processor\" xmlns:rec=\"http://www.eclipse.org/smila/record\"\r\n  
xmlns:bpel=\"http://docs.oasis-open.org/wsbpel/2.0/process/executable\">\r\n\r\n  <import location=\"processor.wsdl\" 
namespace=\"http://www.eclipse.org/smila/processor\"\r\n    importType=\"http://schemas.xmlsoap.org/wsdl/\" />\r\n\r\n
  <partnerLinks>\r\n    <partnerLink name=\"Pipeline\" partnerLinkType=\"proc:ProcessorPartnerLinkType\" myRole=\"service\" 
/>\r\n  </partnerLinks>\r\n\r\n  <extensions>\r\n    <extension namespace=\"http://www.eclipse.org/smila/processor\" 
mustUnderstand=\"no\" />\r\n  </extensions>\r\n\r\n  <variables>\r\n    <variable name=\"request\" messageType=
\"proc:ProcessorMessage\" />\r\n  </variables>\r\n\r\n  <sequence name=\"SearchPipeline\">\r\n    <receive name=\"start\" 
partnerLink=\"Pipeline\" portType=\"proc:ProcessorPortType\"\r\n      operation=\"process\" variable=\"request\" 
createInstance=\"yes\" />\r\n\r\n    <extensionActivity>\r\n      <proc:invokePipelet name=\"search\">\r\n        
<proc:pipelet class=\"org.eclipse.smila.lucene.pipelets.LuceneSearchPipelet\" />\r\n        <proc:variables input=\"request\" 
/>\r\n        <proc:configuration>\r\n          <rec:Val key=\"indexname\">test_index</rec:Val>\r\n        
</proc:configuration>\r\n      </proc:invokePipelet>\r\n    </extensionActivity>\r\n\r\n    <extensionActivity>\r\n
      <proc:invokePipelet name=\"highlight results\">\r\n        <proc:pipelet class=
\"org.eclipse.smila.search.highlighting.HighlightingPipelet\" />\r\n        <proc:variables input=\"request\" />\r\n    
    <proc:configuration>\r\n          <rec:Map key=\"highlightingTransformers\">\r\n            <rec:Map key=\"Content\">
\r\n              <rec:Val key=\"name\">Sentence</rec:Val>\r\n              <rec:Val key=\"MaxLength\">300</rec:Val>\r
\n              <rec:Val key=\"MarkupPrefix\"><b></rec:Val>\r\n              <rec:Val key=\"MarkupSuffix\"></b>
</rec:Val>\r\n              <rec:Val key=\"MaxHLElements\">999</rec:Val>\r\n              <rec:Val key=\"MaxSucceedingCharacters
\">30</rec:Val>\r\n              <rec:Val key=\"SucceedingCharacters\">...</rec:Val>\r\n              <rec:Val 
key=\"SortAlgorithm\">Occurrence</rec:Val>\r\n              <rec:Val key=\"TextHandling\">ReturnSnipplet</rec:Val>\r\n 
           </rec:Map>\r\n          </rec:Map>\r\n        </proc:configuration>\r\n      </proc:invokePipelet>\r\n    
</extensionActivity>\r\n\r\n    <reply name=\"end\" partnerLink=\"Pipeline\" portType=\"proc:ProcessorPortType\" 
operation=\"process\"\r\n      variable=\"request\" />\r\n    <exit />\r\n  </sequence>\r\n</process>\r\n"
}

Whoops, what's this? It's a bit awkward to read because newline and double-quote characters are printed in their JSON-escaped form. In a browser with a [JSONView] extension installed it looks quite readable, e.g. in Firefox:

That's better. We see that the object has a "readOnly" flag set to true, because it is one of the pipeline predefined in the system configuration. If it was a custom pipeline defined via the API, there would be a "timestamp" attribute at the end of the object.

Now we try to execute this pipeline:

> GET /smila/pipeline/SearchPipeline/process/
{
  "message" : "Cannot process an empty record."
}
> POST /smila/pipeline/SearchPipeline/process/
{
  "message" : "Cannot process an empty record."
}

So this did not work, as expected. In the response headers you can see the error code returned:

> POST /smila/pipeline/SearchPipeline/process/ -v
...
< HTTP/1.1 400 Bad Request
...
{
  "message" : "Cannot process an empty record."
}

So we add a query attribute:

> POST /smila/pipeline/SearchPipeline/process/ '{ "query": "SMILA" }'
...
< HTTP/1.1 500 Server Error
...
{
  "message" : "Error processing BPEL workflow SearchPipeline: Invocation of pipeline element SearchPipeline/search failed: Error processing message SearchPipeline-7afe423a-749c-4492-aa66-38ce37dba672\ncaused by: Invocation of pipeline element SearchPipeline/search failed: Error processing message SearchPipeline-7afe423a-749c-4492-aa66-38ce37dba672\ncaused by: no single value for required parameter QueryAttribute"
}

This time the LuceneSearchPipelet complains about a missing parameter. So let's add it:

> POST /smila/pipeline/SearchPipeline/process/ '{ 
    "query": "SMILA", 
    "QueryAttribute": "Content" }' 
{
  "query" : "SMILA",
  "QueryAttribute" : "Content",
  "_recordid" : "SearchPipeline-5c2d3f3f-1e56-4362-aa4c-74aa5fa9d6e8",
  "count" : 58,
  "indexSize" : 115,
  "records" : [ {
    "_recordid" : "feeds:<Uri=tag:search.twitter.com,2005:69739397733560320>",
    "_source" : "feeds",
    "_weight" : 0.84
  }, {
    "_recordid" : "feeds:<Uri=tag:search.twitter.com,2005:69960011966713856>",
    "_source" : "feeds",
    "_weight" : 0.73
  }, 
  ...
  ] }
}

A successful search ... but we probably want to see a bit more information. So we add values for parameter "resultAttributes":

> POST /smila/pipeline/SearchPipeline/process/ '{ 
  "query": "SMILA", 
  "QueryAttribute": "Content", 
  "resultAttributes": [ "Title", "Author", "LastModifiedDate" ] }' 
{
  "query" : "SMILA",
  "QueryAttribute" : "Content",
  "resultAttributes" : [ "Title", "Author", "LastModifiedDate" ],
  "_recordid" : "SearchPipeline-2201c83a-a9b6-4cfe-a62e-6ec8ffe113ca",
  "count" : 58,
  "indexSize" : 115,
  "records" : [ {
    "_recordid" : "feeds:<Uri=tag:search.twitter.com,2005:69739397733560320>",
    "_source" : "feeds",
    "_weight" : 0.84,
    "Title" : "@AbrarAlAdwani smila 3alech. Latwaswiseen",
    "Author" : "dee_the_bee (Dalalee Boland)",
    "LastModifiedDate" : "2011-05-15T14:22:22.000+0200"
  }, {
    "_recordid" : "feeds:<Uri=tag:search.twitter.com,2005:69960011966713856>",
    "_source" : "feeds",
    "_weight" : 0.73,
    "Title" : "Finally found something to smila bout :)",
    "Author" : "MIGUELALMENDRAL (Miguel FG Almendral)",
    "LastModifiedDate" : "2011-05-16T04:59:01.000+0200"
  },
  ...
  ] }
}

Awesome! In the same way you can add more parameters as you need:

> POST /smila/pipeline/SearchPipeline/process/ '{ 
    "query": "SMILA", 
    "QueryAttribute": "Content", 
    "resultAttributes": [ "Title", "Author", "LastModifiedDate" ], 
    "maxcount": 1, "offset": 42, "highlight": "Content" }' 
{
  "query" : "SMILA",
  "QueryAttribute" : "Content",
  "resultAttributes" : [ "Title", "Author", "LastModifiedDate" ],
  "maxcount" : 1,
  "offset" : 42,
  "highlight" : "Content",
  "_recordid" : "SearchPipeline-f2a53434-36b1-4e25-a716-f8c02fba5ecd",
  "count" : 58,
  "indexSize" : 115,
  "records" : [ {
    "_recordid" : "feeds:<Uri=http://www.eclipse.org/forums/index.php/mv/msg/206311/660699/#msg_660699>",
    "_source" : "feeds",
    "_weight" : 0.19,
    "Title" : "Re: New SMILA tryout",
    "Author" : "Andreas Weber",
    "_highlight" : {
      "Content" : {
        "text" : "<br />\n&#62;<br />\n&#62; I've done all standard steps as we do when creating a new<br />\n&#62; <b>SMILA</b>-workspace (and as described...-workspace (and as described on the <b>SMILA</b> website - dont know the<br... website - dont know the<br />\n&#62; exact name right now):<br />\n&#62;<br />\n&#62;<br />\n&#62; check out from eclipse SVN<br />\n&#62; add a new Target Platform with path to the <b>SMILA</b> bundles and to eclipse<br..."
      }
    }
  } ]
}

You can also use GET and URL parameters to invoke the pipeline, just enter something like this in the address line of your favorite browser:

http://localhost:8080/smila/pipeline/SearchPipeline/process?query=SMILA&QueryAttribute=Content

However, this gets inconvenient if you want to add lots of parameters and attributes.