Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "SMILA/Documentation/JobManagerFirstExample"
(→Finish the Job) |
(→Finish the Job Run) |
||
(27 intermediate revisions by the same user not shown) | |||
Line 2: | Line 2: | ||
{{note| | {{note| | ||
− | See [[SMILA/Documentation/Processing/ | + | See [[SMILA/Documentation/Processing/JSON REST API for BPEL pipelines]] on how to use the JSON ReST API}}. |
− | == | + | === Use a Workflow === |
− | + | You could create your own asynchronous workflow, but we use the "indexUpdate" workflow that is already provided with SMILA. It uses the BPEL pipelines from the standard configuration to add and delete index documents. | |
<source lang="javascript"> | <source lang="javascript"> | ||
− | + | GET /smila/jobmanager/workflows/indexUpdate/ | |
+ | |||
+ | HTTP/1.x 200 OK | ||
+ | |||
{ | { | ||
"name": "indexUpdate", | "name": "indexUpdate", | ||
"parameters": | "parameters": | ||
{ | { | ||
− | " | + | "pipelineRunBulkSize": "20" |
}, | }, | ||
"startAction": | "startAction": | ||
Line 22: | Line 25: | ||
{ | { | ||
"insertedRecords": "addBucket", | "insertedRecords": "addBucket", | ||
− | "deletedRecords": " | + | "deletedRecords": "deleteBucket" |
} | } | ||
}, | }, | ||
Line 28: | Line 31: | ||
[ | [ | ||
{ | { | ||
− | "worker": " | + | "worker": "pipelineProcessor", |
"parameters": | "parameters": | ||
{ | { | ||
Line 39: | Line 42: | ||
}, | }, | ||
{ | { | ||
− | "worker": " | + | "worker": "pipelineProcessor", |
"parameters": | "parameters": | ||
{ | { | ||
Line 46: | Line 49: | ||
"input": | "input": | ||
{ | { | ||
− | "input": " | + | "input": "deleteBucket" |
} | } | ||
} | } | ||
] | ] | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
} | } | ||
</source> | </source> | ||
Line 70: | Line 63: | ||
POST /smila/jobmanager/jobs/ | POST /smila/jobmanager/jobs/ | ||
{ | { | ||
− | "name": " | + | "name": "exampleIndexUpdate", |
"workflow": "indexUpdate", | "workflow": "indexUpdate", | ||
"parameters": | "parameters": | ||
Line 79: | Line 72: | ||
</source> | </source> | ||
− | You get a | + | You get a reponse: |
<source lang="javascript"> | <source lang="javascript"> | ||
{ | { | ||
− | "name": " | + | "name": "exampleIndexUpdate", |
"timestamp": "2011-08-15T16:20:34.337+0200", | "timestamp": "2011-08-15T16:20:34.337+0200", | ||
− | "url": "http://localhost:8080/smila/jobmanager/jobs/ | + | "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/" |
} | } | ||
</source> | </source> | ||
Line 94: | Line 87: | ||
<source lang="javascript"> | <source lang="javascript"> | ||
− | POST /smila/jobmanager/jobs/ | + | POST /smila/jobmanager/jobs/exampleIndexUpdate/ |
</source> | </source> | ||
Line 102: | Line 95: | ||
{ | { | ||
"jobId": "20110815-162046851752", | "jobId": "20110815-162046851752", | ||
− | "url": "http://localhost:8080/smila/jobmanager/jobs/ | + | "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/" |
} | } | ||
</source> | </source> | ||
− | === Add a | + | We will need the URL from this response later to finish the job run. |
+ | |||
+ | === Add a Document === | ||
<source lang="javascript"> | <source lang="javascript"> | ||
− | POST /smila/job/ | + | POST /smila/job/exampleIndexUpdate/record/ |
{ | { | ||
"_recordid": "test.html", | "_recordid": "test.html", | ||
Line 120: | Line 115: | ||
</source> | </source> | ||
− | + | Flush the bulk: | |
<source lang="javascript"> | <source lang="javascript"> | ||
− | POST /smila/job/ | + | POST /smila/job/exampleIndexUpdate/record/ |
</source> | </source> | ||
Line 132: | Line 127: | ||
"workflowRunId": "1", | "workflowRunId": "1", | ||
"jobRunId": "20110815-162046851752", | "jobRunId": "20110815-162046851752", | ||
− | "url": "http://localhost:8080/ | + | "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/1/" |
} | } | ||
</source> | </source> | ||
− | After a | + | After a while (about a minute) the document can be found in the sample search site [http://localhost:8080/SMILA/search http://localhost:8080/SMILA/search]. Hint: search for "first". |
− | === Delete a | + | === Delete a Document === |
<source lang="javascript"> | <source lang="javascript"> | ||
− | DELETE /smila/job/ | + | DELETE /smila/job/exampleIndexUpdate/record/?_recordid=test.html |
</source> | </source> | ||
− | + | Flush the bulk: | |
<source lang="javascript"> | <source lang="javascript"> | ||
− | POST /smila/job/ | + | POST /smila/job/exampleIndexUpdate/record/ |
</source> | </source> | ||
Line 156: | Line 151: | ||
"workflowRunId": "2", | "workflowRunId": "2", | ||
"jobRunId": "20110815-162046851752", | "jobRunId": "20110815-162046851752", | ||
− | "url": "http://localhost:8080/ | + | "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/2/" |
} | } | ||
</source> | </source> | ||
− | After a | + | After a while (about a minute), the search should not return any results anymore. |
− | === Finish the Job === | + | === Finish the Job Run === |
Look up the URL from the response of the start-job request and add "finish" to get the path for this POST request: | Look up the URL from the response of the start-job request and add "finish" to get the path for this POST request: | ||
<source lang="javascript"> | <source lang="javascript"> | ||
− | POST /smila/jobmanager/jobs/ | + | POST /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/finish/ |
</source> | </source> | ||
− | The response will be empty, but you should get a response code of 202. Finally you can | + | The response will be empty, but you should get a response code of 202. |
+ | |||
+ | Finally you can request statistics about this job run: | ||
<source lang="javascript"> | <source lang="javascript"> | ||
− | GET /smila/jobmanager/jobs/ | + | GET /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/ |
</source> | </source> | ||
Line 183: | Line 180: | ||
"finishTime": "2011-08-15T16:52:18.714+0200", | "finishTime": "2011-08-15T16:52:18.714+0200", | ||
"jobId": "20110815-162046851752", | "jobId": "20110815-162046851752", | ||
− | " | + | "mode": "STANDARD", |
"startTime": "2011-08-15T16:20:46.920+0200", | "startTime": "2011-08-15T16:20:46.920+0200", | ||
"state": "SUCCEEDED", | "state": "SUCCEEDED", |
Latest revision as of 12:06, 26 January 2012
This is a simple walkthrough on index building using the new JSON ReST APIs and the job management.
.
Contents
Use a Workflow
You could create your own asynchronous workflow, but we use the "indexUpdate" workflow that is already provided with SMILA. It uses the BPEL pipelines from the standard configuration to add and delete index documents.
GET /smila/jobmanager/workflows/indexUpdate/ HTTP/1.x 200 OK { "name": "indexUpdate", "parameters": { "pipelineRunBulkSize": "20" }, "startAction": { "worker": "bulkbuilder", "output": { "insertedRecords": "addBucket", "deletedRecords": "deleteBucket" } }, "actions": [ { "worker": "pipelineProcessor", "parameters": { "pipelineName": "AddPipeline" }, "input": { "input": "addBucket" } }, { "worker": "pipelineProcessor", "parameters": { "pipelineName": "DeletePipeline" }, "input": { "input": "deleteBucket" } } ] }
Create a Job
Now we have to create a job that uses this workflow:
POST /smila/jobmanager/jobs/ { "name": "exampleIndexUpdate", "workflow": "indexUpdate", "parameters": { "tempStore": "tempStore" } }
You get a reponse:
{ "name": "exampleIndexUpdate", "timestamp": "2011-08-15T16:20:34.337+0200", "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/" }
Start a Job Run
Now this job has to be started:
POST /smila/jobmanager/jobs/exampleIndexUpdate/
The response is:
{ "jobId": "20110815-162046851752", "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/" }
We will need the URL from this response later to finish the job run.
Add a Document
POST /smila/job/exampleIndexUpdate/record/ { "_recordid": "test.html", "_source": "handcrafted", "Title": "Hello Job World!", "Content": "This is the first document added to an SMILA index using the new job management", "MimeType": "text/plain", "Size": 42 }
Flush the bulk:
POST /smila/job/exampleIndexUpdate/record/
For both requests the response should be similar to:
{ "workflowRunId": "1", "jobRunId": "20110815-162046851752", "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/1/" }
After a while (about a minute) the document can be found in the sample search site http://localhost:8080/SMILA/search. Hint: search for "first".
Delete a Document
DELETE /smila/job/exampleIndexUpdate/record/?_recordid=test.html
Flush the bulk:
POST /smila/job/exampleIndexUpdate/record/
Again, you get a response for both request like this:
{ "workflowRunId": "2", "jobRunId": "20110815-162046851752", "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/2/" }
After a while (about a minute), the search should not return any results anymore.
Finish the Job Run
Look up the URL from the response of the start-job request and add "finish" to get the path for this POST request:
POST /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/finish/
The response will be empty, but you should get a response code of 202.
Finally you can request statistics about this job run:
GET /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/
and get:
{ "endTime": "2011-08-15T16:52:18.726+0200", "finishTime": "2011-08-15T16:52:18.714+0200", "jobId": "20110815-162046851752", "mode": "STANDARD", "startTime": "2011-08-15T16:20:46.920+0200", "state": "SUCCEEDED", "workflowRuns": { "activeWorkflowRunCount": 0, "canceledWorkflowRunCount": 0, "failedWorkflowRunCount": 0, "startedWorkflowRunCount": 2, "successfulWorkflowRunCount": 2 }, "tasks": { "canceledTaskCount": 0, "createdTaskCount": 4, "failedAfterRetryTaskCount": 0, "failedWithoutRetryTaskCount": 0, "obsoleteTaskCount": 0, "retriedAfterErrorTaskCount": 0, "retriedAfterTimeoutTaskCount": 0, "successfulTaskCount": 4 }, "worker": { ... }, "jobDefinition": { ... } }