Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "SMILA/Documentation/JobManagerFirstExample"
m (→Use a workflow) |
(→Finish the Job Run) |
||
(19 intermediate revisions by the same user not shown) | |||
Line 2: | Line 2: | ||
{{note| | {{note| | ||
− | See [[SMILA/Documentation/Processing/ | + | See [[SMILA/Documentation/Processing/JSON REST API for BPEL pipelines]] on how to use the JSON ReST API}}. |
=== Use a Workflow === | === Use a Workflow === | ||
Line 17: | Line 17: | ||
"parameters": | "parameters": | ||
{ | { | ||
− | " | + | "pipelineRunBulkSize": "20" |
}, | }, | ||
"startAction": | "startAction": | ||
Line 31: | Line 31: | ||
[ | [ | ||
{ | { | ||
− | "worker": " | + | "worker": "pipelineProcessor", |
"parameters": | "parameters": | ||
{ | { | ||
Line 42: | Line 42: | ||
}, | }, | ||
{ | { | ||
− | "worker": " | + | "worker": "pipelineProcessor", |
"parameters": | "parameters": | ||
{ | { | ||
Line 63: | Line 63: | ||
POST /smila/jobmanager/jobs/ | POST /smila/jobmanager/jobs/ | ||
{ | { | ||
− | "name": " | + | "name": "exampleIndexUpdate", |
"workflow": "indexUpdate", | "workflow": "indexUpdate", | ||
"parameters": | "parameters": | ||
Line 76: | Line 76: | ||
<source lang="javascript"> | <source lang="javascript"> | ||
{ | { | ||
− | "name": " | + | "name": "exampleIndexUpdate", |
"timestamp": "2011-08-15T16:20:34.337+0200", | "timestamp": "2011-08-15T16:20:34.337+0200", | ||
− | "url": "http://localhost:8080/smila/jobmanager/jobs/ | + | "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/" |
} | } | ||
</source> | </source> | ||
Line 87: | Line 87: | ||
<source lang="javascript"> | <source lang="javascript"> | ||
− | POST /smila/jobmanager/jobs/ | + | POST /smila/jobmanager/jobs/exampleIndexUpdate/ |
</source> | </source> | ||
Line 95: | Line 95: | ||
{ | { | ||
"jobId": "20110815-162046851752", | "jobId": "20110815-162046851752", | ||
− | "url": "http://localhost:8080/smila/jobmanager/jobs/ | + | "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/" |
} | } | ||
</source> | </source> | ||
Line 101: | Line 101: | ||
We will need the URL from this response later to finish the job run. | We will need the URL from this response later to finish the job run. | ||
− | === Add a | + | === Add a Document === |
<source lang="javascript"> | <source lang="javascript"> | ||
− | POST /smila/job/ | + | POST /smila/job/exampleIndexUpdate/record/ |
{ | { | ||
"_recordid": "test.html", | "_recordid": "test.html", | ||
Line 115: | Line 115: | ||
</source> | </source> | ||
− | + | Flush the bulk: | |
<source lang="javascript"> | <source lang="javascript"> | ||
− | POST /smila/job/ | + | POST /smila/job/exampleIndexUpdate/record/ |
</source> | </source> | ||
Line 127: | Line 127: | ||
"workflowRunId": "1", | "workflowRunId": "1", | ||
"jobRunId": "20110815-162046851752", | "jobRunId": "20110815-162046851752", | ||
− | "url": "http://localhost:8080/ | + | "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/1/" |
} | } | ||
</source> | </source> | ||
− | After a | + | After a while (about a minute) the document can be found in the sample search site [http://localhost:8080/SMILA/search http://localhost:8080/SMILA/search]. Hint: search for "first". |
− | === Delete a | + | === Delete a Document === |
<source lang="javascript"> | <source lang="javascript"> | ||
− | DELETE /smila/job/ | + | DELETE /smila/job/exampleIndexUpdate/record/?_recordid=test.html |
</source> | </source> | ||
− | + | Flush the bulk: | |
<source lang="javascript"> | <source lang="javascript"> | ||
− | POST /smila/job/ | + | POST /smila/job/exampleIndexUpdate/record/ |
</source> | </source> | ||
Line 151: | Line 151: | ||
"workflowRunId": "2", | "workflowRunId": "2", | ||
"jobRunId": "20110815-162046851752", | "jobRunId": "20110815-162046851752", | ||
− | "url": "http://localhost:8080/ | + | "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/2/" |
} | } | ||
</source> | </source> | ||
− | After a | + | After a while (about a minute), the search should not return any results anymore. |
=== Finish the Job Run === | === Finish the Job Run === | ||
Line 162: | Line 162: | ||
<source lang="javascript"> | <source lang="javascript"> | ||
− | POST /smila/jobmanager/jobs/ | + | POST /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/finish/ |
</source> | </source> | ||
The response will be empty, but you should get a response code of 202. | The response will be empty, but you should get a response code of 202. | ||
− | Finally you can | + | Finally you can request statistics about this job run: |
<source lang="javascript"> | <source lang="javascript"> | ||
− | GET /smila/jobmanager/jobs/ | + | GET /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/ |
</source> | </source> | ||
Line 180: | Line 180: | ||
"finishTime": "2011-08-15T16:52:18.714+0200", | "finishTime": "2011-08-15T16:52:18.714+0200", | ||
"jobId": "20110815-162046851752", | "jobId": "20110815-162046851752", | ||
− | " | + | "mode": "STANDARD", |
"startTime": "2011-08-15T16:20:46.920+0200", | "startTime": "2011-08-15T16:20:46.920+0200", | ||
"state": "SUCCEEDED", | "state": "SUCCEEDED", |
Latest revision as of 12:06, 26 January 2012
This is a simple walkthrough on index building using the new JSON ReST APIs and the job management.
.
Contents
Use a Workflow
You could create your own asynchronous workflow, but we use the "indexUpdate" workflow that is already provided with SMILA. It uses the BPEL pipelines from the standard configuration to add and delete index documents.
GET /smila/jobmanager/workflows/indexUpdate/ HTTP/1.x 200 OK { "name": "indexUpdate", "parameters": { "pipelineRunBulkSize": "20" }, "startAction": { "worker": "bulkbuilder", "output": { "insertedRecords": "addBucket", "deletedRecords": "deleteBucket" } }, "actions": [ { "worker": "pipelineProcessor", "parameters": { "pipelineName": "AddPipeline" }, "input": { "input": "addBucket" } }, { "worker": "pipelineProcessor", "parameters": { "pipelineName": "DeletePipeline" }, "input": { "input": "deleteBucket" } } ] }
Create a Job
Now we have to create a job that uses this workflow:
POST /smila/jobmanager/jobs/ { "name": "exampleIndexUpdate", "workflow": "indexUpdate", "parameters": { "tempStore": "tempStore" } }
You get a reponse:
{ "name": "exampleIndexUpdate", "timestamp": "2011-08-15T16:20:34.337+0200", "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/" }
Start a Job Run
Now this job has to be started:
POST /smila/jobmanager/jobs/exampleIndexUpdate/
The response is:
{ "jobId": "20110815-162046851752", "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/" }
We will need the URL from this response later to finish the job run.
Add a Document
POST /smila/job/exampleIndexUpdate/record/ { "_recordid": "test.html", "_source": "handcrafted", "Title": "Hello Job World!", "Content": "This is the first document added to an SMILA index using the new job management", "MimeType": "text/plain", "Size": 42 }
Flush the bulk:
POST /smila/job/exampleIndexUpdate/record/
For both requests the response should be similar to:
{ "workflowRunId": "1", "jobRunId": "20110815-162046851752", "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/1/" }
After a while (about a minute) the document can be found in the sample search site http://localhost:8080/SMILA/search. Hint: search for "first".
Delete a Document
DELETE /smila/job/exampleIndexUpdate/record/?_recordid=test.html
Flush the bulk:
POST /smila/job/exampleIndexUpdate/record/
Again, you get a response for both request like this:
{ "workflowRunId": "2", "jobRunId": "20110815-162046851752", "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/2/" }
After a while (about a minute), the search should not return any results anymore.
Finish the Job Run
Look up the URL from the response of the start-job request and add "finish" to get the path for this POST request:
POST /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/finish/
The response will be empty, but you should get a response code of 202.
Finally you can request statistics about this job run:
GET /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/
and get:
{ "endTime": "2011-08-15T16:52:18.726+0200", "finishTime": "2011-08-15T16:52:18.714+0200", "jobId": "20110815-162046851752", "mode": "STANDARD", "startTime": "2011-08-15T16:20:46.920+0200", "state": "SUCCEEDED", "workflowRuns": { "activeWorkflowRunCount": 0, "canceledWorkflowRunCount": 0, "failedWorkflowRunCount": 0, "startedWorkflowRunCount": 2, "successfulWorkflowRunCount": 2 }, "tasks": { "canceledTaskCount": 0, "createdTaskCount": 4, "failedAfterRetryTaskCount": 0, "failedWithoutRetryTaskCount": 0, "obsoleteTaskCount": 0, "retriedAfterErrorTaskCount": 0, "retriedAfterTimeoutTaskCount": 0, "successfulTaskCount": 4 }, "worker": { ... }, "jobDefinition": { ... } }