Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/JettyHttpServer"

(Usage: fixed hint to add org.eclipse.smila.http.server[.util] and made it more prominent)
(Adding Request Handlers)
 
(37 intermediate revisions by 3 users not shown)
Line 32: Line 32:
 
           <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
 
           <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
 
             <Set name="host"><Property name="jetty.host" /></Set>
 
             <Set name="host"><Property name="jetty.host" /></Set>
             <Set name="port"><Property name="jetty.port" default="8080"/></Set>
+
             <Set name="port"><Property name="jetty.port" /></Set>
 
             <Set name="maxIdleTime">300000</Set>
 
             <Set name="maxIdleTime">300000</Set>
 
             <Set name="Acceptors">2</Set>
 
             <Set name="Acceptors">2</Set>
 
             <Set name="statsOn">false</Set>
 
             <Set name="statsOn">false</Set>
            <Set name="confidentialPort">8443</Set>
 
 
    <Set name="lowResourcesConnections">20000</Set>
 
    <Set name="lowResourcesConnections">20000</Set>
 
    <Set name="lowResourcesMaxIdleTime">5000</Set>
 
    <Set name="lowResourcesMaxIdleTime">5000</Set>
Line 47: Line 46:
 
     <!-- =========================================================== -->
 
     <!-- =========================================================== -->
 
     <Set name="handler">
 
     <Set name="handler">
       <New id="DefaultHandler" class="org.eclipse.jetty.server.handler.DefaultHandler"/>
+
       <New id="Contexts" class="org.eclipse.jetty.server.handler.HandlerCollection" />
 
     </Set>
 
     </Set>
  
Line 62: Line 61:
 
</source>
 
</source>
  
The default configuration is not especially useful by itself. It basically configures the server to listen at port 8080, adds a default handler responding with HTTP status 404 (NOT FOUND) if no other handler was found to handle the request, and lists the available handlers.
+
The default configuration is not especially useful by itself. It basically configures the server to listen at port 8080, adds a default handler responding with HTTP status 404 (NOT FOUND) if no other handler was found to handle the request.
  
 
For more details on all the configuration properties used here, refer to the [[Jetty|Jetty documentation]] next door.
 
For more details on all the configuration properties used here, refer to the [[Jetty|Jetty documentation]] next door.
  
=== Usage ===
+
==== Setting the HTTP port ====
 +
 
 +
Usually, the port on which the Jetty server will listen is configured via the [[SMILA/Documentation/Bundle_org.eclipse.smila.clusterconfig|ClusterConfig service]]. This means that in a standard SMILA setup you set the HTTP port in the configuration file <tt>configuration/org.eclipse.smila.clusterconfig.simple/clusterconfig.json</tt> which will set the configuration property <tt>jetty.port</tt> on startup of the HTTP server and thus override the default value given for this property in <tt>jetty.xml</tt>.
 +
 
 +
Example snippet from the <tt>clusterconfig.json</tt> file:
 +
<source lang="javascript">
 +
{
 +
  ...,
 +
  "services":{
 +
    "smila":{
 +
      "httpPort":8080
 +
    }
 +
  },
 +
  ...
 +
}
 +
</source>
 +
 
 +
You usually use this property in jetty.xml to configure the "Connector" object:
 +
<source lang="xml">
 +
<Call name="addConnector">
 +
  <Arg>
 +
    <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
 +
      <Set name="host"><SystemProperty name="jetty.host" default="localhost" /></Set>
 +
      <Set name="port"><Property name="jetty.port" default="8080" /></Set>
 +
  <!-- ... -->
 +
</Call>   
 +
</source>
 +
 
 +
{{Tip|
 +
To be exact: the service name will be determined by the <tt>serviceName</tt> property of the HTTP service component definition for the service implementation <tt>org.eclipse.smila.http.server.internal.HttpServiceImpl</tt>. If (and only if) this property and the <tt>ClusterConfigService</tt> are both available, the <tt>ClusterConfigService</tt> service will be called to determine the configured http port for the service by calling its <tt>int getHttpPort(final String serviceName)</tt> method. If the result is not <tt>-1</tt> it will be set as the HTTP port for the HttpService. Otherwise the port as defined in the <tt>jetty.xml</tt> file will be used.
 +
}}
 +
 
 +
The ClusterConfigService allows to define a second port number named "httpSecurePort" which the HTTP server sets as configuration property <tt>jetty.securePort</tt>. If the cluster configuration does not contain value for the secure port, but a value for the standard port, the HTTP service sets <tt>jetty.securePort</tt> to <tt>jetty.port + 1</tt>. You can use this <tt>jetty.securePort</tt> in jetty.xml just like the <tt>jetty.port</tt> for example to configure a second connector for HTTPS access, for example:
 +
 
 +
Example snippet from the <tt>clusterconfig.json</tt> file:
 +
<source lang="javascript">
 +
{
 +
  ...,
 +
  "services":{
 +
    "smila":{
 +
      "httpPort":8080
 +
      "httpSecurePort":18443
 +
    }
 +
  },
 +
  ...
 +
}
 +
</source>
 +
 
 +
Example snippet from <tt>jetty.xml</tt>:
 +
<source lang="xml">
 +
  <Call name="addConnector">
 +
    <Arg>
 +
      <New class="org.eclipse.jetty.server.ssl.SslSelectChannelConnector">
 +
        <Arg>
 +
          <New class="org.eclipse.jetty.http.ssl.SslContextFactory">
 +
            <!-- configure key store for SSL certificate -->
 +
          </New>
 +
        </Arg>
 +
        <Set name="host"><SystemProperty name="jetty.host" /></Set>
 +
        <Set name="port"><Property name="jetty.securePort" default="8443" /></Set>
 +
    ...
 +
  </Call>
 +
</source>
 +
 
 +
For details about configuration of the SSL certificate, please refer to the [[Jetty/Howto/Configure_SSL|Jetty documentation]].
 +
 
 +
==== Configuring Authentication ====
 +
 
 +
You can add authentication to the SMILA HTTP server by configuring LoginServices and SecurityHandlers in jetty.xml. For example:
 +
 
 +
<source lang="xml">
 +
<Call name="addBean">
 +
  <Arg>
 +
    <New id="LoginService" class="org.eclipse.jetty.security.HashLoginService">
 +
      <Set name="name">SMILA Realm</Set>
 +
      <Set name="config">password.properties</Set>
 +
      <Set name="refreshInterval">0</Set>
 +
    </New>
 +
  </Arg>
 +
</Call>
 +
 
 +
<Set name="handler">
 +
  <New id="security" class="org.eclipse.jetty.security.ConstraintSecurityHandler">
 +
    <Set name="Authenticator">
 +
      <New class="org.eclipse.jetty.security.authentication.BasicAuthenticator" />
 +
    </Set>
 +
    <Set name="ConstraintMappings">
 +
      <Array type="org.eclipse.jetty.security.ConstraintMapping">
 +
        <Item>
 +
          <New class="org.eclipse.jetty.security.ConstraintMapping">
 +
            <Set name="PathSpec">/*</Set>
 +
            <Set name="Constraint">
 +
              <New class="org.eclipse.jetty.util.security.Constraint">
 +
                <Set name="Name">auth</Set>
 +
                <Set name="Authenticate">true</Set>
 +
                <Set name="Roles">
 +
                  <Array type="java.lang.String">
 +
                  <Item>user</Item></Array>
 +
                </Set>
 +
              </New>
 +
            </Set>
 +
          </New>
 +
        </Item>
 +
      </Array>
 +
    </Set>
 +
    <Set name="handler">
 +
      <New id="Contexts" class="org.eclipse.jetty.server.handler.HandlerCollection" />
 +
    </Set>
 +
  </New>
 +
</Set>
 +
</source>
 +
 
 +
This enables BASIC authentication for the complete SMILA HTTP server with users, passwords and roles configured in a file "password.properties". For the format of this file or for a description of other LoginServices provided by Jetty see the [[Jetty/Tutorial/Realms|Jetty documentation]].
 +
 
 +
SMILA provides an own simple LoginService that allows to specify one user/password combination in jetty.xml immediately, for example. The password can be specified as plain text, MD5 hash or encrypted, see [[Jetty/Tutorial/Realms|Jetty documentation]] for details on format and how to create them.  The name property will be the realm string reported by the webserver at login.
 +
 
 +
<source lang="xml">
 +
<Call name="addBean">
 +
  <Arg>
 +
    <New id="LoginService" class="org.eclipse.smila.http.server.jetty.SingleUserLoginService">
 +
      <Set name="name">SMILA Realm</Set>
 +
      <Set name="user"><Property name="jetty.user" default="user" /></Set>
 +
      <Set name="password"><Property name="jetty.password" default="1234" /></Set>
 +
    </New>
 +
  </Arg>
 +
</Call>
 +
</source>
 +
 
 +
<tt>jetty.user</tt> and <tt>jetty.password</tt> are two more configuration properties which the HTTP service gets from the cluster configuration: To use this, you must set the property "httpAuthUser" to the user name, and "httpAuthPasswordHash" to the MD5 hash of the password hash in <tt>clusterconfig.json</tt>:
 +
 
 +
<source lang="xml">
 +
{
 +
  "httpAuthUser": "johndoe",
 +
  "httpAuthPasswordHash": "3858f62230ac3c915f300c664312c63f" // MD5 hash of "foobar"
 +
  "services": {
 +
    "smila": {
 +
      "httpPort": 8080
 +
    }
 +
  },
 +
  ...
 +
}
 +
</source>
 +
 
 +
The password hash must be appropriate for the selected authentication scheme (see the setting the "Authenticator" of the SecurityHandler in jetty.xml). You can create MD5 hashes for example here:
 +
* for BASIC authentication: [http://hashgenerator.de/ http://hashgenerator.de/]
 +
* for DIGEST authentication: [http://www.askapache.com/online-tools/htpasswd-generator/ http://www.askapache.com/online-tools/htpasswd-generator/]: enter Username, Password and Realm (DigestDomain is irrelevant), select Encryption Algorithm "digest" and Authentication Scheme "Digest", click "Generate HTPASSWD" and copy the hash part of the "digest Algorihm:" text field.
 +
 
 +
=== Adding Request Handlers ===
  
 
Handlers can be added using two different approaches: Either by extending the <tt>jetty.xml</tt> file, or by registering different kinds of OSGi services that are referenced by the HTTP server service and are registered at startup.
 
Handlers can be added using two different approaches: Either by extending the <tt>jetty.xml</tt> file, or by registering different kinds of OSGi services that are referenced by the HTTP server service and are registered at startup.
  
 
Before we dive into the details, first some important advices:
 
Before we dive into the details, first some important advices:
* To implement functionality, you will probably provide new classes in your own bundle. If the HTTP server needs to instantiate these classes (e.g. servlets used by a web application), you must register your bundle as a "buddy" to the server bundle. So, if you have problems deploying your own code and if you get class loading related errors, first check if your <tt>MANIFEST.MF</tt> contains this line:
+
* If the jetty.xml configuration contains a HandlerCollection with ID "Contexts", all OSGI request handler services will be added to this collection. Otherwise SMILA will create a new HandlerCollection, adds all OSGI services and the configured root handler to this collection and replaces the configured root handler by this new collection. This means: for best control over the Jetty handler tree you should take care to always include a HandlerCollection with ID "Contexts" at the desired position in the tree (cf. all examples on this page or the default jetty.xml in the SMILA distribution).
 +
* To implement functionality, you will probably provide new classes in your own bundle. If the HTTP server needs to instantiate these classes (e.g. servlets used by a web application deployed via jetty.xml), you must register your bundle as a "buddy" to the server bundle. So, if you have problems deploying your own code and if you get class loading related errors, first check if your <tt>MANIFEST.MF</tt> contains this line:
 
<source lang="text">
 
<source lang="text">
 
Eclipse-RegisterBuddy: org.eclipse.smila.http.server
 
Eclipse-RegisterBuddy: org.eclipse.smila.http.server
 
</source>
 
</source>
* You also must add these two imports to the manifest, otherwise the handlers will not be registered with the http server, despite being instantiated:  
+
* Bundles that provide classes that are to be called from Jetty (OSGi services as request handlers, Servlets, etc), must also contain these two Import-Package lines in the manifest (even if they con't actually use classes from these packages), otherwise the HTTP server will not be able to access them:
 
<source lang='text'>
 
<source lang='text'>
 
org.eclipse.smila.http.server, org.eclipse.smila.http.server.util
 
org.eclipse.smila.http.server, org.eclipse.smila.http.server.util
Line 110: Line 257:
 
         </Item>
 
         </Item>
 
         <Item>
 
         <Item>
           <New class="org.eclipse.jetty.server.handler.DefaultHandler"/>
+
           <New id="Contexts" class="org.eclipse.jetty.server.handler.HandlerCollection" />
 
         </Item>
 
         </Item>
 
       </Array>
 
       </Array>
Line 119: Line 266:
  
 
It registers a standard web application located at <tt>configuration/org.eclipse.smila.search.servlet/webapp</tt>. Note that you must also specify the ''defaultsDescriptor'' property because the embedded Jetty cannot find one at the default location.
 
It registers a standard web application located at <tt>configuration/org.eclipse.smila.search.servlet/webapp</tt>. Note that you must also specify the ''defaultsDescriptor'' property because the embedded Jetty cannot find one at the default location.
 +
 +
'''Example:''' Adding an extra resource handler to serve static files. This makes files in directory <tt>/home/smila/Images</tt> accessible at <tt>http://.../Images/</tt>:
 +
 +
<source lang="xml">
 +
<Set name="handler">
 +
  <New class="org.eclipse.jetty.server.handler.HandlerList">
 +
    <Set name="handlers">
 +
      <Array type="org.eclipse.jetty.server.Handler">
 +
 +
        <!-- ... other handlers ... -->
 +
 +
        <Item>
 +
          <New class="org.eclipse.jetty.server.handler.ContextHandler">
 +
            <Set name="contextPath">/Images</Set>
 +
            <Set name="handler">
 +
              <New class="org.eclipse.jetty.server.handler.ResourceHandler">
 +
                <Set name="directoriesListed">true</Set>
 +
                <Set name="resourceBase">/home/smila/Images</Set>
 +
              </New>
 +
          </Set>
 +
        </New>
 +
      </Item>
 +
      <Item> <!-- this one is always at the end of the list -->
 +
        <New id="Contexts" class="org.eclipse.jetty.server.handler.HandlerCollection" />
 +
      </Item>
 +
    </Array>
 +
  </New>
 +
</Set>
 +
</source>
  
 
==== HttpHandler services ====
 
==== HttpHandler services ====
Line 176: Line 352:
 
* JsonHttpHandler: implements the HttpHandler interface
 
* JsonHttpHandler: implements the HttpHandler interface
 
* JsonRequestHandler: implements the RequestHandler interface and extends ARequestHandler
 
* JsonRequestHandler: implements the RequestHandler interface and extends ARequestHandler
* JsonBulkRequestHandler: implements the RequestHandler interface and extends ARequestHandler. Can take multiple JSON objects as input, but does not produce a result object. [Note: starting with release 0.9.0 a result may be produced by the finish method and will be returned in the response as JSON.]
+
* JsonBulkRequestHandler: implements the RequestHandler interface and extends ARequestHandler. Can take multiple JSON objects as input, but (normally) does not produce a result object. [Note: a result may be produced by the finish method and will be returned in the response as JSON.]
  
 
The first two handler expect a single record as input to be processed (of course, the record can be empty). The record is either parsed from a JSON request body (in POST or PUT requests) or constructed from the request parameters (in a GET request) and then a <tt>process()</tt> method implemented by the subclass is invoked with this record to do the actual processing. The result may be a single object, preferably a SMILA Record or Any object.
 
The first two handler expect a single record as input to be processed (of course, the record can be empty). The record is either parsed from a JSON request body (in POST or PUT requests) or constructed from the request parameters (in a GET request) and then a <tt>process()</tt> method implemented by the subclass is invoked with this record to do the actual processing. The result may be a single object, preferably a SMILA Record or Any object.
  
The latter handler expects a JSON "bulk" as its input. A bulk is a set of JSON Objects, each one printed on a single line (i.e. no newline or linefeed characters) and seperated by newline characters. The bulk is parsed and sent to a <tt>process()</tt> method implemented in a subclass one by one, so this handler makes it possible to push a lot of records into SMILA in a single request with high performance and low memory usage. After the complete bulk has been processed successfully, a <tt>finish()</tt> method implemented by the subclass is called to finalize the processing if necessary.
+
The latter handler expects a JSON "bulk" as its input. A bulk is a set of JSON Objects, each one printed on a single line (i.e. no newline or linefeed characters) and seperated by newline characters. The bulk is parsed and sent to a <tt>process()</tt> method implemented in a subclass one by one, so this handler makes it possible to push a lot of records into SMILA in a single request with high performance and low memory usage. After the complete bulk has been processed successfully, a <tt>finish()</tt> method implemented by the subclass is called to finalize the processing if necessary. A result produced by the finish method will be returned as a response.
  
 
The first handler is meant to be managed by the Http service immediately (register as HttpHandler) while the latter two are managed by a RequestDispatcher service (register them as RequestHandlers).
 
The first handler is meant to be managed by the Http service immediately (register as HttpHandler) while the latter two are managed by a RequestDispatcher service (register them as RequestHandlers).
Line 187: Line 363:
 
* <tt>isValidMethod</tt>: Checks if the HTTP method is allowed for this handler. If not, a METHOD-NOT-ALLOWED status is returned to the client. By default only POST is allowed.
 
* <tt>isValidMethod</tt>: Checks if the HTTP method is allowed for this handler. If not, a METHOD-NOT-ALLOWED status is returned to the client. By default only POST is allowed.
 
* <tt>getSuccessStatus</tt>: HTTP status to return after successful processing. Default is ACCEPTED for the bulk handler and OK for the others.
 
* <tt>getSuccessStatus</tt>: HTTP status to return after successful processing. Default is ACCEPTED for the bulk handler and OK for the others.
* <tt>writeResultObject</tt>: Add additional Object-to-JSON-serialization code if you do not return a SMILA Record or Any object or something else that can be serialized using a standard [http://wiki.fasterxml.com/ObjectMapper Jackson ObjectMapper]. The latter usually works fine for Java objects like Collections or Maps or Beans.
+
* <tt>writeResultObject</tt>: Add additional Object-to-JSON-serialization code if you do not return a SMILA Record or Any object or something else that can be serialized using a standard [http://wiki.fasterxml.com/ObjectMapper Jackson ObjectMapper]. The latter usually works fine for Java objects like Collections or Maps or Beans.
 
* <tt>getErrorStatus</tt>: HTTP status to return after failed processing, depending on the action exception. See the default implementation for the predefined mapping from exception type to status code.
 
* <tt>getErrorStatus</tt>: HTTP status to return after failed processing, depending on the action exception. See the default implementation for the predefined mapping from exception type to status code.
 +
 +
===== Attachments =====
 +
 +
<tt>JsonHttpHandler</tt> and <tt>JsonRequestHandler</tt> support attachments for input records in POST requests, too. Such records must be sent as "multipart" requests, where the first part contains the record metadata as JSON, followed by binary parts.
 +
 +
For example, to submit a record with an attachment to a job (which is the main use case for sending attachments to SMILA), you can use the [http://hc.apache.org/httpcomponents-client-ga/index.html Apache HttpClient 4.x] (not included with SMILA, however):
 +
 +
<source lang="java">
 +
import org.apache.http.HttpResponse;
 +
import org.apache.http.HttpStatus;
 +
import org.apache.http.client.HttpClient;
 +
import org.apache.http.client.methods.HttpPost;
 +
import org.apache.http.entity.mime.MultipartEntity;
 +
import org.apache.http.entity.mime.content.ByteArrayBody;
 +
import org.apache.http.entity.mime.content.StringBody;
 +
import org.apache.http.impl.client.DefaultHttpClient;
 +
...
 +
String jsonMetadata = ...;
 +
byte[] attachment = ...;
 +
HttpClient client = new DefaultHttpClient();
 +
MultipartEntity multiPartMessage = new MultipartEntity();
 +
StringBody jsonPart = new StringBody(jsonMetadata,
 +
                                    "application/json",
 +
                                    Charset.forName("UTF-8"));
 +
multiPartMessage.addPart("metadata", jsonPart);
 +
                      // the actual name of the metadata part is irrelevant
 +
ByteArrayBody attachmentPart = new ByteArrayBody(attachment,
 +
                                                "application/octet-stream",
 +
                                                null); // no filename necessary
 +
multiPartMessage.addPart("content", attachmentPart);
 +
                      // part name is the attachment name.
 +
HttpPost request = new HttpPost("http://localhost:8080/smila/job/" + jobName + "/record");
 +
request.setEntity(multiPartMessage);
 +
HttpResponse response = client.execute(request);
 +
...
 +
</source>
 +
 +
The HTTP request than looks similar to this:
 +
 +
<source lang="text">
 +
POST /smila/job/jobname/record
 +
 +
Content-Length: 12345
 +
Content-Type: multipart/form-data; boundary=UMFOMEI8xxUic4zcDix9Utn4ORyVTcS3
 +
 +
--UMFOMEI8xxUic4zcDix9Utn4ORyVTcS3
 +
Content-Disposition: form-data; name="metadata"
 +
Content-Type: application/json; charset=UTF-8
 +
Content-Transfer-Encoding: 8bit
 +
 +
{
 +
"_recordid": "...",
 +
// more attributes
 +
}
 +
--UMFOMEI8xxUic4zcDix9Utn4ORyVTcS3
 +
Content-Disposition: form-data; name="content"
 +
Content-Type: application/octet-stream
 +
Content-Transfer-Encoding: binary
 +
 +
... attachment content ...
 +
 +
--UMFOMEI8xxUic4zcDix9Utn4ORyVTcS3--
 +
 +
</source>
 +
 +
The <tt>HttpClient</tt> lib provides additional [http://hc.apache.org/httpcomponents-client-ga/httpmime/apidocs/org/apache/http/entity/mime/content/package-summary.html <tt>ContentBody</tt>] implementations for adding parts from <tt>InputStream</tt>s or <tt>File</tt>s immediately instead of having to load them in memory first. Of course, you can also add multiple attachments to a single request, just add more parts with different attachment names.
 +
 +
We have also introduced a size limit for incoming records to protect SMILA against OutOfMemoryErrors by client requests. The size limit can be configured in <tt>configuration/org.eclipse.smila.http.server/httpserver.properties</tt>, the default value is 1 GiB:
 +
 +
<source lang="text">
 +
# maximum size of request records (including attachments) at the ReST API
 +
# for JSON handlers that load the complete request data in memory (using JsonHttpUtils.convertRequest())
 +
http.json.maxRequestRecordSize=1g
 +
</source>
 +
 +
Values can use "k" (KiB), "m" (MiB), "g" (GiB) suffixes for better readability. The value describes the maximum size of the complete request, i.e. JSON metadata plus all attachments. Bigger requests are rejected with response code <tt>400 BAD REQUEST</tt>. The limit does not concern handlers derived from <tt>JsonBulkRequestHandler</tt> or non-Json-handlers, as it should still be possible to implement "streaming handlers" that do not need to load the complete request in memory.
  
 
==== JettyHandler services ====
 
==== JettyHandler services ====
Line 243: Line 495:
 
===== WebappContextService =====
 
===== WebappContextService =====
  
Creates a handler for a web application. The example from the test bundle goes like this:  
+
Creates a handler for a web application. The example from the test bundle is as follows:  
  
 
<source lang="xml">
 
<source lang="xml">

Latest revision as of 05:54, 18 July 2014

Configuration and Usage of the Jetty HTTP server embedded in SMILA

Overview

The embedding of the Jetty server is implemented in bundle org.eclipse.smila.http.server. When the bundle is activated it starts the OSGi service org.eclipse.smila.http.server.HttpService which in turn creates a Jetty server from a configuration file, adds request handlers provided by other OSGi services and starts the server.

Configuration

To configure the embedded Jetty server, place a file named jetty.xml in the configuration directory configuration/org.eclipse.smila.http.server. If the configuration area does not contain such a file, a default file provided by the HTTP server bundle itself is used. It's probably the most simple file possible:

<Configure id="Server" class="org.eclipse.jetty.server.Server">
 
    <!-- =========================================================== -->
    <!-- Server Thread Pool                                          -->
    <!-- =========================================================== -->
    <Set name="ThreadPool">
      <!-- Default queued blocking threadpool -->
      <New class="org.eclipse.jetty.util.thread.QueuedThreadPool">
        <Set name="minThreads">10</Set>
        <Set name="maxThreads">200</Set>
        <Set name="detailedDump">false</Set>
      </New>
    </Set>
 
    <!-- =========================================================== -->
    <!-- Set connectors                                              -->
    <!-- =========================================================== -->
 
    <Call name="addConnector">
      <Arg>
          <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
            <Set name="host"><Property name="jetty.host" /></Set>
            <Set name="port"><Property name="jetty.port" /></Set>
            <Set name="maxIdleTime">300000</Set>
            <Set name="Acceptors">2</Set>
            <Set name="statsOn">false</Set>
	    <Set name="lowResourcesConnections">20000</Set>
	    <Set name="lowResourcesMaxIdleTime">5000</Set>
          </New>
      </Arg>
    </Call>
 
    <!-- =========================================================== -->
    <!-- Set handler Collection Structure                            --> 
    <!-- =========================================================== -->
    <Set name="handler">
      <New id="Contexts" class="org.eclipse.jetty.server.handler.HandlerCollection" />
    </Set>
 
    <!-- =========================================================== -->
    <!-- extra options                                               -->
    <!-- =========================================================== -->
    <Set name="stopAtShutdown">true</Set>
    <Set name="sendServerVersion">true</Set>
    <Set name="sendDateHeader">true</Set>
    <Set name="gracefulShutdown">1000</Set>
    <Set name="dumpAfterStart">false</Set>
    <Set name="dumpBeforeStop">false</Set>
</Configure>

The default configuration is not especially useful by itself. It basically configures the server to listen at port 8080, adds a default handler responding with HTTP status 404 (NOT FOUND) if no other handler was found to handle the request.

For more details on all the configuration properties used here, refer to the Jetty documentation next door.

Setting the HTTP port

Usually, the port on which the Jetty server will listen is configured via the ClusterConfig service. This means that in a standard SMILA setup you set the HTTP port in the configuration file configuration/org.eclipse.smila.clusterconfig.simple/clusterconfig.json which will set the configuration property jetty.port on startup of the HTTP server and thus override the default value given for this property in jetty.xml.

Example snippet from the clusterconfig.json file:

{
  ...,
  "services":{
    "smila":{
      "httpPort":8080
    }
  },
  ...
}

You usually use this property in jetty.xml to configure the "Connector" object:

<Call name="addConnector">
  <Arg>
    <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
      <Set name="host"><SystemProperty name="jetty.host" default="localhost" /></Set>
      <Set name="port"><Property name="jetty.port" default="8080" /></Set>
  <!-- ... -->
</Call>
Idea.png

To be exact: the service name will be determined by the serviceName property of the HTTP service component definition for the service implementation org.eclipse.smila.http.server.internal.HttpServiceImpl. If (and only if) this property and the ClusterConfigService are both available, the ClusterConfigService service will be called to determine the configured http port for the service by calling its int getHttpPort(final String serviceName) method. If the result is not -1 it will be set as the HTTP port for the HttpService. Otherwise the port as defined in the jetty.xml file will be used.



The ClusterConfigService allows to define a second port number named "httpSecurePort" which the HTTP server sets as configuration property jetty.securePort. If the cluster configuration does not contain value for the secure port, but a value for the standard port, the HTTP service sets jetty.securePort to jetty.port + 1. You can use this jetty.securePort in jetty.xml just like the jetty.port for example to configure a second connector for HTTPS access, for example:

Example snippet from the clusterconfig.json file:

{
  ...,
  "services":{
    "smila":{
      "httpPort":8080
      "httpSecurePort":18443
    }
  },
  ...
}

Example snippet from jetty.xml:

  <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ssl.SslSelectChannelConnector">
        <Arg>
          <New class="org.eclipse.jetty.http.ssl.SslContextFactory">
            <!-- configure key store for SSL certificate -->
          </New>
        </Arg>
        <Set name="host"><SystemProperty name="jetty.host" /></Set>
        <Set name="port"><Property name="jetty.securePort" default="8443" /></Set>
    ...
  </Call>

For details about configuration of the SSL certificate, please refer to the Jetty documentation.

Configuring Authentication

You can add authentication to the SMILA HTTP server by configuring LoginServices and SecurityHandlers in jetty.xml. For example:

<Call name="addBean">
  <Arg>
    <New id="LoginService" class="org.eclipse.jetty.security.HashLoginService">
      <Set name="name">SMILA Realm</Set>
      <Set name="config">password.properties</Set>
      <Set name="refreshInterval">0</Set>
    </New>
  </Arg>
</Call>
 
<Set name="handler">
  <New id="security" class="org.eclipse.jetty.security.ConstraintSecurityHandler">
    <Set name="Authenticator">
      <New class="org.eclipse.jetty.security.authentication.BasicAuthenticator" />
    </Set>
    <Set name="ConstraintMappings">
      <Array type="org.eclipse.jetty.security.ConstraintMapping">
        <Item>
          <New class="org.eclipse.jetty.security.ConstraintMapping">
            <Set name="PathSpec">/*</Set>
            <Set name="Constraint">
              <New class="org.eclipse.jetty.util.security.Constraint">
                <Set name="Name">auth</Set>
                <Set name="Authenticate">true</Set>
                <Set name="Roles">
                  <Array type="java.lang.String">
                  <Item>user</Item></Array>
                </Set>
              </New>
            </Set>
          </New>
        </Item>
      </Array>
    </Set>
    <Set name="handler">
      <New id="Contexts" class="org.eclipse.jetty.server.handler.HandlerCollection" />
    </Set>
  </New>
</Set>

This enables BASIC authentication for the complete SMILA HTTP server with users, passwords and roles configured in a file "password.properties". For the format of this file or for a description of other LoginServices provided by Jetty see the Jetty documentation.

SMILA provides an own simple LoginService that allows to specify one user/password combination in jetty.xml immediately, for example. The password can be specified as plain text, MD5 hash or encrypted, see Jetty documentation for details on format and how to create them. The name property will be the realm string reported by the webserver at login.

<Call name="addBean">
  <Arg>
    <New id="LoginService" class="org.eclipse.smila.http.server.jetty.SingleUserLoginService">
      <Set name="name">SMILA Realm</Set>
      <Set name="user"><Property name="jetty.user" default="user" /></Set>
      <Set name="password"><Property name="jetty.password" default="1234" /></Set>
    </New>
  </Arg>
</Call>

jetty.user and jetty.password are two more configuration properties which the HTTP service gets from the cluster configuration: To use this, you must set the property "httpAuthUser" to the user name, and "httpAuthPasswordHash" to the MD5 hash of the password hash in clusterconfig.json:

{
  "httpAuthUser": "johndoe",
  "httpAuthPasswordHash": "3858f62230ac3c915f300c664312c63f" // MD5 hash of "foobar"
  "services": {
    "smila": {
      "httpPort": 8080
    }
  },
  ...
}

The password hash must be appropriate for the selected authentication scheme (see the setting the "Authenticator" of the SecurityHandler in jetty.xml). You can create MD5 hashes for example here:

Adding Request Handlers

Handlers can be added using two different approaches: Either by extending the jetty.xml file, or by registering different kinds of OSGi services that are referenced by the HTTP server service and are registered at startup.

Before we dive into the details, first some important advices:

  • If the jetty.xml configuration contains a HandlerCollection with ID "Contexts", all OSGI request handler services will be added to this collection. Otherwise SMILA will create a new HandlerCollection, adds all OSGI services and the configured root handler to this collection and replaces the configured root handler by this new collection. This means: for best control over the Jetty handler tree you should take care to always include a HandlerCollection with ID "Contexts" at the desired position in the tree (cf. all examples on this page or the default jetty.xml in the SMILA distribution).
  • To implement functionality, you will probably provide new classes in your own bundle. If the HTTP server needs to instantiate these classes (e.g. servlets used by a web application deployed via jetty.xml), you must register your bundle as a "buddy" to the server bundle. So, if you have problems deploying your own code and if you get class loading related errors, first check if your MANIFEST.MF contains this line:
Eclipse-RegisterBuddy: org.eclipse.smila.http.server
  • Bundles that provide classes that are to be called from Jetty (OSGi services as request handlers, Servlets, etc), must also contain these two Import-Package lines in the manifest (even if they con't actually use classes from these packages), otherwise the HTTP server will not be able to access them:
org.eclipse.smila.http.server, org.eclipse.smila.http.server.util
  • The Jetty server must be stopped and restarted to register additional handlers. Therefore, if you use OSGi services to register functionality in the server, you should take care that the HTTP server bundle is started in a higher run level than all bundles providing handler services. This is the reason why the HTTP server is started on the highest run level of all bundles in the SMILA application. A nice side effect of this is that you can check from outside if the startup of SMILA has finished: If you can connect to the HTTP server, this means that SMILA is up and running:
..., \
org.eclipse.smila.http.server@5:start
  • Also, you should not add handler services to the HTTP server bundle itself using Declarative Services, because the start order of DS services in a single bundle is not deterministic.

Deployment via jetty.xml

The straightforward way is to use the jetty.xml configuration file itself to add handlers for different URL contexts. In theory, it should be possible to do everything you can normally do in this configuration file, see the Jetty documentation for details. However, there may be some class loading fun ahead.

The default configuration file used in the SMILA application replaces the "handler" section of the default configuration to deploy a simple web application for search at http://localhost:8080/SMILA/search:

<!-- =========================================================== -->
<!-- Set handler Collection Structure                            --> 
<!-- =========================================================== -->
<Set name="handler">
  <New class="org.eclipse.jetty.server.handler.HandlerList">
    <Set name="handlers">
      <Array type="org.eclipse.jetty.server.Handler">
        <Item>
          <New class="org.eclipse.jetty.webapp.WebAppContext">
            <Set name="contextPath">/SMILA</Set>
            <Set name="resourceBase">configuration/org.eclipse.smila.search.servlet/webapp</Set>
            <Set name="descriptor">configuration/org.eclipse.smila.search.servlet/webapp/WEB-INF/web.xml</Set>
            <Set name="defaultsDescriptor">configuration/org.eclipse.smila.http.server/webdefault.xml</Set>
            <Set name="parentLoaderPriority">true</Set>
          </New>
        </Item>
        <Item>
          <New id="Contexts" class="org.eclipse.jetty.server.handler.HandlerCollection" />
        </Item>
      </Array>
    </Set>
  </New>
</Set>

It registers a standard web application located at configuration/org.eclipse.smila.search.servlet/webapp. Note that you must also specify the defaultsDescriptor property because the embedded Jetty cannot find one at the default location.

Example: Adding an extra resource handler to serve static files. This makes files in directory /home/smila/Images accessible at http://.../Images/:

<Set name="handler">
  <New class="org.eclipse.jetty.server.handler.HandlerList">
    <Set name="handlers">
      <Array type="org.eclipse.jetty.server.Handler">
 
        <!-- ... other handlers ... -->
 
        <Item>
          <New class="org.eclipse.jetty.server.handler.ContextHandler">
            <Set name="contextPath">/Images</Set>
            <Set name="handler">
              <New class="org.eclipse.jetty.server.handler.ResourceHandler">
                <Set name="directoriesListed">true</Set>
                <Set name="resourceBase">/home/smila/Images</Set>
              </New>
           </Set>
        </New>
      </Item>
      <Item> <!-- this one is always at the end of the list -->
        <New id="Contexts" class="org.eclipse.jetty.server.handler.HandlerCollection" />
      </Item>
    </Array>
  </New>
</Set>

HttpHandler services

org.eclipse.smila.http.server.HttpHandler is the most primitive SMILA specific interface for OSGi services for handling HTTP requests. It defines two methods:

  • String getRootContextPath(): return the context path (i.e. the part of the URL after the "http://<host>:<port>" stuff) for which this handler should be invoked. This is used by the HTTP server to select an appropriate handler for a request. The value must be a valid Jetty context path: The handler should be invoked for all requests to URI paths that start with this value.
  • void handle(final HttpExchange exchange) throws IOException: Actually handle the request. The HttpExchange object gives access to the HTTP method, request URI, request headers and input stream as well as response headers and output stream.

See org.eclipse.smila.http.server.test.MockHttpHandler in the HTTP server test bundle for the most simple implementation of this services. OSGI-INF/httphandler.xml shows how to start such a server using DS.

RequestDispatcher and RequestHandler services

The org.eclipse.smila.http.server.util.RequestDispatcher is a HttpHandler implementation that uses org.eclipse.smila.http.server.util.RequestHandler services to handle requests for its root context. RequestHandlers are HttpHandlers but cannot register for handling URIs containing prefixes. Instead, the dispatcher calls matches(String requestUri) and the RequestHandler can check programmatically if it should handle this request. Also, the dispatcher can add and remove request handlers without having to restart the complete HTTP server. To use a RequestDispatcher, you must start it using a DS descriptor. E.g. see the example in the ...http.server.test bundle in OSGI-INF/requestdispatcher.xml:

<?xml version="1.0" encoding="UTF-8"?>
<component name="org.eclipse.smila.http.server.util.RequestDispatcher" immediate="true">
    <implementation class="org.eclipse.smila.http.server.util.RequestDispatcher" />
    <service>
       <provide interface="org.eclipse.smila.http.server.HttpHandler"/>
    </service>
    <reference
      name="RequestHandlers"
      interface="org.eclipse.smila.http.server.util.RequestHandler"
      bind="addRequestHandler"
      unbind="removeRequestHandler"
      cardinality="0..n"
      policy="dynamic"
      target="(rootContextPath=/dispatch)" 
      />    
</component>

This starts a dispatcher for the root context "/dispatch", i.e. it tries to handle all requests to http://host:port/dispatch... and uses all RequestHandler services that have a property rootContextPath set to "/dispatch". You can add this description to your own bundles to start several dispatchers for different root contexts, just be sure to import the bundles org.eclipse.smila.http.server and org.eclipse.smila.http.server.util even if they are not needed by the actual code.

We provide also a base class for RequestHandlers named org.eclipse.smila.http.server.util.ARequestHandler which implements the matches() method matching the URI part after the root context path using a regular expression string read from its own component properties specified in the DS file. E.g. in the test bundle we have OSGI-INF/requesthandler.xml:

<?xml version="1.0" encoding="UTF-8"?>
<component name="org.eclipse.smila.http.server.test.MockRequestHandler" immediate="true">
    <implementation class="org.eclipse.smila.http.server.test.MockRequestHandler" />    
    <service>
       <provide interface="org.eclipse.smila.http.server.util.RequestHandler"/>
    </service>
    <property name="rootContextPath" type="String" value="/dispatch"/>
    <property name="uriPattern" type="String" value="/handle/([^/]+)/?$"/>    
</component>

This starts a simple RequestHandler service managed by the dispatcher described above and reacts on requests to http://host:port/dispatch/handle/<string-without-further-slashes>. Also, the base class provides methods to extract the parts of the request URI matched by the groups in the regular expression.

JSON Handlers

SMILA provides some base classes that make it quite easy to write handlers that receive requests and produce results using JSON. JSON is a very lightweight data format that makes interchange much more efficient than using XML. Moreover it is easy to interpret in Javascript based web user interfaces. Currently the following base classes are available in package org.eclipse.smila.http.server.json:

  • JsonHttpHandler: implements the HttpHandler interface
  • JsonRequestHandler: implements the RequestHandler interface and extends ARequestHandler
  • JsonBulkRequestHandler: implements the RequestHandler interface and extends ARequestHandler. Can take multiple JSON objects as input, but (normally) does not produce a result object. [Note: a result may be produced by the finish method and will be returned in the response as JSON.]

The first two handler expect a single record as input to be processed (of course, the record can be empty). The record is either parsed from a JSON request body (in POST or PUT requests) or constructed from the request parameters (in a GET request) and then a process() method implemented by the subclass is invoked with this record to do the actual processing. The result may be a single object, preferably a SMILA Record or Any object.

The latter handler expects a JSON "bulk" as its input. A bulk is a set of JSON Objects, each one printed on a single line (i.e. no newline or linefeed characters) and seperated by newline characters. The bulk is parsed and sent to a process() method implemented in a subclass one by one, so this handler makes it possible to push a lot of records into SMILA in a single request with high performance and low memory usage. After the complete bulk has been processed successfully, a finish() method implemented by the subclass is called to finalize the processing if necessary. A result produced by the finish method will be returned as a response.

The first handler is meant to be managed by the Http service immediately (register as HttpHandler) while the latter two are managed by a RequestDispatcher service (register them as RequestHandlers).

Other methods you may want to override in subclasses to customize the behavior depending on HTTP method used to invoke the handler, the actual request URI, input and output objects and exceptions throwm by the process method:

  • isValidMethod: Checks if the HTTP method is allowed for this handler. If not, a METHOD-NOT-ALLOWED status is returned to the client. By default only POST is allowed.
  • getSuccessStatus: HTTP status to return after successful processing. Default is ACCEPTED for the bulk handler and OK for the others.
  • writeResultObject: Add additional Object-to-JSON-serialization code if you do not return a SMILA Record or Any object or something else that can be serialized using a standard Jackson ObjectMapper. The latter usually works fine for Java objects like Collections or Maps or Beans.
  • getErrorStatus: HTTP status to return after failed processing, depending on the action exception. See the default implementation for the predefined mapping from exception type to status code.
Attachments

JsonHttpHandler and JsonRequestHandler support attachments for input records in POST requests, too. Such records must be sent as "multipart" requests, where the first part contains the record metadata as JSON, followed by binary parts.

For example, to submit a record with an attachment to a job (which is the main use case for sending attachments to SMILA), you can use the Apache HttpClient 4.x (not included with SMILA, however):

import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.mime.MultipartEntity;
import org.apache.http.entity.mime.content.ByteArrayBody;
import org.apache.http.entity.mime.content.StringBody;
import org.apache.http.impl.client.DefaultHttpClient;
...
String jsonMetadata = ...;
byte[] attachment = ...;
HttpClient client = new DefaultHttpClient();
MultipartEntity multiPartMessage = new MultipartEntity();
StringBody jsonPart = new StringBody(jsonMetadata, 
                                     "application/json", 
                                     Charset.forName("UTF-8"));
multiPartMessage.addPart("metadata", jsonPart);
                      // the actual name of the metadata part is irrelevant
ByteArrayBody attachmentPart = new ByteArrayBody(attachment, 
                                                 "application/octet-stream", 
                                                 null); // no filename necessary
multiPartMessage.addPart("content", attachmentPart); 
                      // part name is the attachment name.
HttpPost request = new HttpPost("http://localhost:8080/smila/job/" + jobName + "/record");
request.setEntity(multiPartMessage);
HttpResponse response = client.execute(request);
...

The HTTP request than looks similar to this:

POST /smila/job/jobname/record
 
Content-Length: 12345
Content-Type: multipart/form-data; boundary=UMFOMEI8xxUic4zcDix9Utn4ORyVTcS3
 
--UMFOMEI8xxUic4zcDix9Utn4ORyVTcS3
Content-Disposition: form-data; name="metadata"
Content-Type: application/json; charset=UTF-8
Content-Transfer-Encoding: 8bit
 
{
	"_recordid": "...",
	// more attributes
}
--UMFOMEI8xxUic4zcDix9Utn4ORyVTcS3
Content-Disposition: form-data; name="content"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
 
... attachment content ...
 
--UMFOMEI8xxUic4zcDix9Utn4ORyVTcS3--

The HttpClient lib provides additional ContentBody implementations for adding parts from InputStreams or Files immediately instead of having to load them in memory first. Of course, you can also add multiple attachments to a single request, just add more parts with different attachment names.

We have also introduced a size limit for incoming records to protect SMILA against OutOfMemoryErrors by client requests. The size limit can be configured in configuration/org.eclipse.smila.http.server/httpserver.properties, the default value is 1 GiB:

# maximum size of request records (including attachments) at the ReST API 
# for JSON handlers that load the complete request data in memory (using JsonHttpUtils.convertRequest())
http.json.maxRequestRecordSize=1g

Values can use "k" (KiB), "m" (MiB), "g" (GiB) suffixes for better readability. The value describes the maximum size of the complete request, i.e. JSON metadata plus all attachments. Bigger requests are rejected with response code 400 BAD REQUEST. The limit does not concern handlers derived from JsonBulkRequestHandler or non-Json-handlers, as it should still be possible to implement "streaming handlers" that do not need to load the complete request in memory.

JettyHandler services

An OSGi service providing org.eclipse.smila.http.JettyHandler can be used to inject original Jetty handlers programmatically into the server. A JettyHandler must implement two methods:

  • org.eclipse.jetty.server.Handler getHandler(): the initialized Jetty handler. If it is not already a ContextHandler, it is wrapped inside one automatically by the HttpService.
  • String getRootContextPath(): the URI path prefix to be handled by this handler.

There is also a base class org.eclipse.smila.http.AJettyHandlerService available which reads the context path value from the DS component context and determines paths into the configuration area if the component context contains a bundle name for which access to the configuration space has been configured.

We have already created three (partly prototypical) implementations of this service. They provide only quite limited means of configuring the Jetty handler themselves, so they are provided rather as examples, meaning you can use them to base your own handlers on. If you require more configuration, you will either have to create your own variant of these services or use the jetty.xml to achieve the desired deployment/configuration.

The exemplary implementations (details follow below) are all contained in org.eclipse.smila.http.server.util.

ResourceHandlerService

Creates a Jetty ResourceHandler which can serve files from the configuration area. The test bundle shows how to use this in OSGI-INF/resourcehandler.xml:

<?xml version="1.0" encoding="UTF-8"?>
<component name="org.eclipse.smila.http.server.util.ResourceHandlerService" immediate="true">
    <implementation class="org.eclipse.smila.http.server.util.ResourceHandlerService" />
    <service>
       <provide interface="org.eclipse.smila.http.server.JettyHandler"/>
    </service>
    <property name="rootContextPath" type="String" value="/resources"/>
    <property name="configBundle" type="String" value="org.eclipse.smila.http.server.test" />
    <property name="resourceBase" type="String" value="resources"/>    
    <property name="welcomeFiles" type="String" value="index.html"/>
</component>

This creates a ResourceHandler which can provide files from configuration/org.eclipse.smila.http.server.test/resources at the URL http://.../resources/..., and returns "index.html" for requests containing only a directory name and no explicit filename in the URL.

ServletContextService

Creates and registers a ServletContextHandler with a single javax.servlet instance already created (so it may be helpful in case of classloading problems, because the servlet instance can be created inside its own bundle context). Again, you can find a simple example in the test bundle at OSGI-INF/servlethandler.xml:

<?xml version="1.0" encoding="UTF-8"?>
<component name="org.eclipse.smila.http.server.util.ServletContextService" immediate="true">
    <implementation class="org.eclipse.smila.http.server.util.ServletContextService" />
    <service>
       <provide interface="org.eclipse.smila.http.server.JettyHandler"/>
    </service>
    <property name="rootContextPath" type="String" value="/servlet"/>
    <property name="servletClassName" type="String" value="org.eclipse.smila.http.server.test.MockServlet" />
    <property name="servletPath" type="String" value="/mock"/>    
</component>

This registers an instance of the named class as a servlet which can be invoked using http://.../servlet/mock.

WebappContextService

Creates a handler for a web application. The example from the test bundle is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<component name="org.eclipse.smila.http.server.util.WebappContextService" immediate="true">
    <implementation class="org.eclipse.smila.http.server.util.WebappContextService" />
    <service>
       <provide interface="org.eclipse.smila.http.server.JettyHandler"/>
    </service>
    <property name="rootContextPath" type="String" value="/webapp"/>
    <property name="configBundle" type="String" value="org.eclipse.smila.http.server.test" />
    <property name="webappDir" type="String" value="webapp"/>    
</component>

This makes the web application available as http://.../webapp in the directory webapp of the configuration area of bundle org.eclipse.smila.http.server.test. The web application must not be assembled into a WAR file. The descriptor file must be located at WEB-INF/web.xml and the configuration directory for the named bundle must also provide a file named webdefault.xml containing default values for the web.xml. An override-web.xml is not supported. See configuration/org.eclipse.smila.http.server.test/ for an exemplary simple setup.

Back to the top