SWIM Broker

Current Version: 1.0
Language: Java

Terms

Term Definition
CWL Common Workflow Language (https://www.commonwl.org/user_guide/)
Scenario Specification Input serialization for scientific model execution in SWIM
Provenance Origin or traceback of a resource


Introduction

The SWIM Broker acts as the main gateway and service orchestrator for single and model-to-model executions of integrated scientific models.

For single run executions, the broker extracts metadata from the SWIM model catalog and determines which model to run based on the model identifier included on the scenario specification.

For model-to-model workflows, the broker orchestrates a set of services needed for: automated workflow planning (AI Planning), serialization to open standards (CWL), workflow execution (model runs), and postprocessing (provenance and results).

Open API

The SWIM Broker Webservice is documented following the OpenAPI 3.0 Specification with Swagger Docs. Access by clicking on the link below:

SWIM Broker - Swagger Docs

Sequence View

This view represents the sequence of processing steps among webservices involved in the automatic generation and execution of a model-to-model workflow.

Primary Representation

Uses Diagram

Catalog

:UI
User interface from which the workflow request is submitted from. Currently available as an Open API documentation form.

orchestrate:swim-broker
The orchestration endpoint of the SWIM broker service. Acts as the main gateway for model-to-model workflows using integrated models in SWIM.

compose:workflow-composer
The workflow composer automates the generation of a model execution plan (workflow) and outputs the specification in JSON format.

serialize:workflow-cwl
The CWL serializer endpoint takes as input a workflow plan in JSON format and transforms it to CWL format.

execute:workflow-cwl
The CWL execution enpoint takes care of invoking and passing data among model execution and transformation services that form part of the workflow plan.

postprocess:swim-merge
The swim-merge endpoint consumes a workflow unique identifier and extracts modeling outputs specified on the initial workflow requests. A provenance trace of each output is included in the result.

Uses View

The following representation depicts the use of external Java packages and decomposition of internal packages in the SWIM Broker project structure. Each package accounts as a folder that holds one or multiple class files. Only explicitly imported dependencies are shown on the diagram, external packages may use additional dependencies.

Primary Representation

Uses Diagram

Internal Packages

  • auth: JSON token validation and authentication.
  • handlers: Core functionalities for the broker service. It includes preprocessing functions for composer input and web service call handlers.
  • mappers: Mapping classes for composer consumption and SWIM-Workflow mappings.
  • db: Database connectivity and handlers.
  • health: Database healthchecks. Provides status of database services.
  • test: Package for unit testing procedures.
  • models/swim: Object-document mappings with SWIM Modeling database (MongoDB)
  • models/workflow: Object-document mappings with WOrkflow database (MongoDB)
  • services/api: Webservice configurations and start up.
  • services/controller: Webservice endpoint specification and application flow.

External Packages

Use Case

As a use case scenario, we present a model-to-model integration of the Water Balance Model 2.0 and the Hydroeconomic Model, both available under the SWIM platform.

For this particular case, the Hydroeconomic Model will require an output generated from the Water Balance. The time series of surface water evaporation generated from the WB model will be consumed by the Hydroeconomic Model as an input. The workflow infrastructure on SWIM automatically generates and executes the workflow plan which includes the sequence of model and transformation services required to generate the final result.

Primary Representation

The representation below illustrates a workflow plan from the proposed use case scenario. It represents an execution flow of modeling and transformation services required to extract the requested output variables.

Use Case

Use Case Sample Files

Note: Refer to the sequence diagram to follow the workflow generation of these files.

1. Workflow Input A workflow specification submitted by the user.
2a. Composer Catalog Preprocessing output: Catalog of models, inputs and outputs. Applies semantic rules for equivalences.
2b. Composer Input Preprocessing output: Specification of unique input ids and output ids requested.
3. Workflow Plan Workflow-composer output: Generates a workflow execution plan in JSON format.
4. CWL Serialization Workflow serializer output: Coberts the workflow execution to CWL format.
5. Workflow Result Provenance trace and model output results of the requested model variables.

Docker Deployment

The SWIM Broker is packaged as a Docker container for easy server deployment. The diagram below shows the docker-compose structure.

Docker Diagram

Containers:

1. swim-broker: The SWIM broker runs a as Java-based restful webservice. The base OS consists of a custom oracle image ready with OpenJDK. The application is setup as a Maven project and uses the Dropwizard Framework as the webservice wrapper.

Volumes:

The mapped with the broker are used mainly for development and debugging purposes.

1. outputs: The outputs volume stores files generated throughout the workflow process such inputs for the workflow composer and model catalogs.
2. logs: The logs volume store application warnings and errors generated from code exceptions.

Environment Settings:

Current environment settings include connectivity to database dependencies and the endpoint URLS for workflow composition, serialization, run and postprocessing (swim-merge).

Database Dependencies:

1. SWIM Modeling: SWIM modeling database from which the broker uses: model and transformation service registries, model parameter and output metadata, and custom executed scenarios.
2. Workflow: The broker stores and uses intermediate workflow data such as: flow metadata, composer catalogs, composer inputs, flow mappings, model run tracking and final flow outputs.
3. Auth Keys: The broker uses secret keys for JWT authentication.
4. SWIM User: The broker uses SWIM user information and credentials.