Working with workspace layers¶

Workspace layers allow you to share data between tasks in a workflow. It's used to share data from a task to subsequent tasks in a same workflow.

In Razorops, every task in a workflow starts in a separate VM environment and runs independently without sharing a common filesystem as they can run in parallel or DAG pattern. Additionally all tasks in a workflow share a common storage which they can add to and remove from at will during execution of steps. We call this common storage a workspace context and it's items as workspace layers.

A workspace layer is a file or collection of files produced and saved into a context during a workflow run. A task can download these layers by attaching itself to the context and download all layers relative to the current working directory.

You can defines steps to persist and attach workspace layers with workspace/persist, workspace/attach steps defined in this guide.

Note

Workspace layers are deleted as soon as the workflow run completes. Tasks that are dependent on a previous tasks's layer must wait for the dependent task to complete successfully.

Example¶

In the following example, two tasks (t1, t2. t2 depends on t1) are defined that shares a common file (hello.txt). Task t2 attaches to the context before using the common file.

tasks:
  t1:
    steps:
    - run: echo hello > hello.txt
    - workspace/persist:
        paths: [hello.txt]

  t2:
    depends: [t1] ## make sure t1 completes first
    steps:
    - workspace/attach
    - run: cat hello.txt
    ...

Persisting layers¶

Use workspace/persist step to add a layer to current workflow context. It supports following fields:

paths: List of file(s) or directories. A path should be relative to current directory. It doesn't support path globbing (like assets/*.png or */**/*.png).

Restoring layers¶

You can restore all of the layers to a task by simply adding workspace/attach step. Attaching a context needs to download and unpack tarballs for every upstream job that persisted data to the current working directory.

Limitations¶

Though it might be tempting to persist most files between the tasks, there is a definite penalty in terms of network utilization, cpu cycles and time. For speedy workflows you should minimise the archive, upload, download and unpack operations.