Pipeline Orchestrator

Overview

The pipeline orchestrator is responsible for:

• parsing a workflow and generating an executable Argo Workflow
• contacting the data providers involved in the workflow
• contacting the compute provider in order to ensure that the local cluster is ready to run the workflow, and is connected to the remote clusters involved in the execution
• starting the execution of the Argo Workflow

When a StartWorkflowRequest is received, the body of the request will first be translated into a graph model (see ToGraph).

The resulting graph will then be translated into an Argo Workflow see graph package. The cwl of each task and the parameters supplied are used to create Argo Templates and containers. Between each Argo Containers, the input and outputs are passed using a S3 container as defined in the StartWorkflowRequest ( see addS3Bucket function). At the end we have an executable Argo Workflow.

Admiralty specific annotations are added to this workflow if multi cluster execution is needed ( see addAnnotation function in run controller)

The compute provider is called to ensure that our local cluster is ready to execute the workflow, and that it is well-connected to all the others clusters involved in the execution ( see compute_provider_client.go)

Starting the workflow is handled by argo_client.go

Prerequisites

• helm
• skaffold
• kubectl

You’ll need an active kubectl context to use this API in skaffold

Usage

Run

skaffold run

Option default-repo need to be added in skaffold command for specify container registry to used.

Dev

skaffold dev

Test

From the project root, run :

go test ./... -cover

Configuration

Server

Environment variables

Api

Execute workflow

URL: /api/pipeline-orchestrator/start

METHOD: POST

Path Params: None

Query Params: None

Request Body: StartWorkflowRequest

Success Response

Code: 200 OK

Content: StartWorkflowResponse

Error Response

Code: 500 Internal Server Error

Or

Code: 422 Unprocessable entity the request body cannot be parsed

Or

Code: 400 Bad Request the request body has been parsed but is invalid