Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/JobManagerFirstExample"

m
(→‎Finish the Job Run)
 
(23 intermediate revisions by 5 users not shown)
Line 2: Line 2:
  
 
{{note|
 
{{note|
−
See [[SMILA/Documentation/Processing/JSON_REST_API]] on information how to use the JSON ReST API}}
+
See [[SMILA/Documentation/Processing/JSON REST API for BPEL pipelines]] on how to use the JSON ReST API}}.
  
−
== Use a workflow ==
+
=== Use a Workflow ===
  
 
You could create your own asynchronous workflow, but we use the "indexUpdate" workflow that is already provided with SMILA. It uses the BPEL pipelines from the standard configuration to add and delete index documents.
 
You could create your own asynchronous workflow, but we use the "indexUpdate" workflow that is already provided with SMILA. It uses the BPEL pipelines from the standard configuration to add and delete index documents.
Line 10: Line 10:
 
<source lang="javascript">
 
<source lang="javascript">
 
GET /smila/jobmanager/workflows/indexUpdate/
 
GET /smila/jobmanager/workflows/indexUpdate/
 +
 +
HTTP/1.x 200 OK
 +
 
{
 
{
 
     "name": "indexUpdate",
 
     "name": "indexUpdate",
 
     "parameters":  
 
     "parameters":  
 
     {
 
     {
−
         "numberOfParallelRecords": "20"
+
         "pipelineRunBulkSize": "20"
 
     },
 
     },
 
     "startAction":
 
     "startAction":
Line 28: Line 31:
 
     [
 
     [
 
         {
 
         {
−
             "worker": "pipelineProcessingWorker",
+
             "worker": "pipelineProcessor",
 
             "parameters":  
 
             "parameters":  
 
             {
 
             {
Line 39: Line 42:
 
         },
 
         },
 
         {
 
         {
−
             "worker": "pipelineProcessingWorker",
+
             "worker": "pipelineProcessor",
 
             "parameters":  
 
             "parameters":  
 
             {
 
             {
Line 52: Line 55:
 
}
 
}
 
</source>
 
</source>
−
 
  
 
=== Create a Job ===
 
=== Create a Job ===
Line 61: Line 63:
 
POST /smila/jobmanager/jobs/
 
POST /smila/jobmanager/jobs/
 
{
 
{
−
     "name": "indexUpdate",
+
     "name": "exampleIndexUpdate",
 
     "workflow": "indexUpdate",
 
     "workflow": "indexUpdate",
 
     "parameters":  
 
     "parameters":  
Line 74: Line 76:
 
<source lang="javascript">
 
<source lang="javascript">
 
{
 
{
−
     "name": "indexUpdate",
+
     "name": "exampleIndexUpdate",
 
     "timestamp": "2011-08-15T16:20:34.337+0200",
 
     "timestamp": "2011-08-15T16:20:34.337+0200",
−
     "url": "http://localhost:8080/smila/jobmanager/jobs/indexUpdate/"
+
     "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/"
 
}
 
}
 
</source>
 
</source>
Line 85: Line 87:
  
 
<source lang="javascript">
 
<source lang="javascript">
−
POST /smila/jobmanager/jobs/indexUpdate/
+
POST /smila/jobmanager/jobs/exampleIndexUpdate/
 
</source>
 
</source>
  
Line 93: Line 95:
 
{
 
{
 
     "jobId": "20110815-162046851752",
 
     "jobId": "20110815-162046851752",
−
     "url": "http://localhost:8080/smila/jobmanager/jobs/indexUpdate/20110815-162046851752/"
+
     "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/"
 
}
 
}
 
</source>
 
</source>
Line 99: Line 101:
 
We will need the URL from this response later to finish the job run.
 
We will need the URL from this response later to finish the job run.
  
−
=== Add a document ===  
+
=== Add a Document ===  
  
 
<source lang="javascript">
 
<source lang="javascript">
−
POST /smila/job/indexUpdate/record/
+
POST /smila/job/exampleIndexUpdate/record/
 
{
 
{
 
   "_recordid": "test.html",
 
   "_recordid": "test.html",
Line 113: Line 115:
 
</source>
 
</source>
  
−
and flush the bulk:
+
Flush the bulk:
  
 
<source lang="javascript">
 
<source lang="javascript">
−
POST /smila/job/indexUpdate/record/
+
POST /smila/job/exampleIndexUpdate/record/
 
</source>
 
</source>
  
Line 125: Line 127:
 
     "workflowRunId": "1",
 
     "workflowRunId": "1",
 
     "jobRunId": "20110815-162046851752",
 
     "jobRunId": "20110815-162046851752",
−
     "url": "http://localhost:8080/ias/jobmanager/jobs/indexUpdate/20110815-162046851752/workflowrun/1/"
+
     "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/1/"
 
}
 
}
 
</source>
 
</source>
  
−
After a short while the document can be found in the sample search site [http://localhost:8080/SMILA/search http://localhost:8080/SMILA/search]. Hint: search for "first".
+
After a while (about a minute) the document can be found in the sample search site [http://localhost:8080/SMILA/search http://localhost:8080/SMILA/search]. Hint: search for "first".
  
−
=== Delete a document ===
+
=== Delete a Document ===
  
 
<source lang="javascript">
 
<source lang="javascript">
−
DELETE /smila/job/indexUpdate/record/?_recordid=test.html
+
DELETE /smila/job/exampleIndexUpdate/record/?_recordid=test.html
 
</source>
 
</source>
  
−
and flush the bulk:
+
Flush the bulk:
  
 
<source lang="javascript">
 
<source lang="javascript">
−
POST /smila/job/indexUpdate/record/
+
POST /smila/job/exampleIndexUpdate/record/
 
</source>
 
</source>
  
Line 149: Line 151:
 
     "workflowRunId": "2",
 
     "workflowRunId": "2",
 
     "jobRunId": "20110815-162046851752",
 
     "jobRunId": "20110815-162046851752",
−
     "url": "http://localhost:8080/ias/jobmanager/jobs/indexUpdate/20110815-162046851752/workflowrun/2/"
+
     "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/2/"
 
}
 
}
 
</source>
 
</source>
  
−
After a short while, the search should not return any results anymore.
+
After a while (about a minute), the search should not return any results anymore.
  
 
=== Finish the Job Run ===
 
=== Finish the Job Run ===
Line 160: Line 162:
  
 
<source lang="javascript">
 
<source lang="javascript">
−
POST /smila/jobmanager/jobs/indexUpdate/20110815-162046851752/finish/
+
POST /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/finish/
 
</source>
 
</source>
  
 
The response will be empty, but you should get a response code of 202.  
 
The response will be empty, but you should get a response code of 202.  
  
−
Finally you can get statistics about this job run:
+
Finally you can request statistics about this job run:
  
 
<source lang="javascript">
 
<source lang="javascript">
−
GET /smila/jobmanager/jobs/indexUpdate/20110815-162046851752/
+
GET /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/
 
</source>
 
</source>
  
Line 178: Line 180:
 
     "finishTime": "2011-08-15T16:52:18.714+0200",
 
     "finishTime": "2011-08-15T16:52:18.714+0200",
 
     "jobId": "20110815-162046851752",
 
     "jobId": "20110815-162046851752",
−
     "runMode": "STANDARD",
+
     "mode": "STANDARD",
 
     "startTime": "2011-08-15T16:20:46.920+0200",
 
     "startTime": "2011-08-15T16:20:46.920+0200",
 
     "state": "SUCCEEDED",
 
     "state": "SUCCEEDED",

Latest revision as of 12:06, 26 January 2012

This is a simple walkthrough on index building using the new JSON ReST APIs and the job management.

Note.png

.

Use a Workflow

You could create your own asynchronous workflow, but we use the "indexUpdate" workflow that is already provided with SMILA. It uses the BPEL pipelines from the standard configuration to add and delete index documents. 

GET /smila/jobmanager/workflows/indexUpdate/
 
HTTP/1.x 200 OK
 
{
    "name": "indexUpdate",
    "parameters": 
    {
        "pipelineRunBulkSize": "20"
    },
    "startAction":
    {
        "worker": "bulkbuilder",
        "output":
        {
            "insertedRecords": "addBucket",
            "deletedRecords": "deleteBucket"
        }
    },
    "actions":
    [
        {
            "worker": "pipelineProcessor",
            "parameters": 
            {
                "pipelineName": "AddPipeline"
            },
            "input":
            {
                "input": "addBucket"
            }
        },
        {
            "worker": "pipelineProcessor",
            "parameters": 
            {
                "pipelineName": "DeletePipeline"
            },
            "input":
            {
                "input": "deleteBucket"
            }
        }
    ]
}

Create a Job

Now we have to create a job that uses this workflow: 

POST /smila/jobmanager/jobs/
{
    "name": "exampleIndexUpdate",
    "workflow": "indexUpdate",
    "parameters": 
    {
        "tempStore": "tempStore"
    }
}

You get a reponse: 

{
    "name": "exampleIndexUpdate",
    "timestamp": "2011-08-15T16:20:34.337+0200",
    "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/"
}

Start a Job Run

Now this job has to be started: 

POST /smila/jobmanager/jobs/exampleIndexUpdate/

The response is: 

{
    "jobId": "20110815-162046851752",
    "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/"
}

We will need the URL from this response later to finish the job run.

Add a Document

POST /smila/job/exampleIndexUpdate/record/
{
  "_recordid": "test.html",
  "_source": "handcrafted",
  "Title": "Hello Job World!",
  "Content": "This is the first document added to an SMILA index using the new job management",
  "MimeType": "text/plain",
  "Size": 42
}

Flush the bulk: 

POST /smila/job/exampleIndexUpdate/record/

For both requests the response should be similar to: 

{
    "workflowRunId": "1",
    "jobRunId": "20110815-162046851752",
    "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/1/"
}

After a while (about a minute) the document can be found in the sample search site http://localhost:8080/SMILA/search. Hint: search for "first".

Delete a Document

DELETE /smila/job/exampleIndexUpdate/record/?_recordid=test.html

Flush the bulk: 

POST /smila/job/exampleIndexUpdate/record/

Again, you get a response for both request like this: 

{
    "workflowRunId": "2",
    "jobRunId": "20110815-162046851752",
    "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/2/"
}

After a while (about a minute), the search should not return any results anymore.

Finish the Job Run

Look up the URL from the response of the start-job request and add "finish" to get the path for this POST request: 

POST /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/finish/

The response will be empty, but you should get a response code of 202.

Finally you can request statistics about this job run: 

GET /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/

and get: 

{
    "endTime": "2011-08-15T16:52:18.726+0200",
    "finishTime": "2011-08-15T16:52:18.714+0200",
    "jobId": "20110815-162046851752",
    "mode": "STANDARD",
    "startTime": "2011-08-15T16:20:46.920+0200",
    "state": "SUCCEEDED",
    "workflowRuns": {
        "activeWorkflowRunCount": 0,
        "canceledWorkflowRunCount": 0,
        "failedWorkflowRunCount": 0,
        "startedWorkflowRunCount": 2,
        "successfulWorkflowRunCount": 2
    },
    "tasks": {
        "canceledTaskCount": 0,
        "createdTaskCount": 4,
        "failedAfterRetryTaskCount": 0,
        "failedWithoutRetryTaskCount": 0,
        "obsoleteTaskCount": 0,
        "retriedAfterErrorTaskCount": 0,
        "retriedAfterTimeoutTaskCount": 0,
        "successfulTaskCount": 4
    },
    "worker": { ... },
    "jobDefinition": { ... }
}
Retrieved from "https://wiki.eclipse.org/index.php?title=SMILA/Documentation/JobManagerFirstExample&oldid=286785"

This page was last modified 12:06, 26 January 2012 by Juergen Schumacher. Based on work by , thomas menzel and Drazen Cindric and others.

Back to the top