YAML Specification¶

This document is a reference for the Razorops YAML configuration keys that are used in the razorops.yaml file.

version¶

version key defines the pipeline definition version. Currently only v2 is the supported values (default: v2).

global¶

global key defines settings common to all tasks in a pipeline. Currently it supports runner configuration and environment variables which are passed to each task. A task can override runner property as well.

global:
  runner:
    containers:
    - image: razorci/ruby:3
    - image: redis
  variables:
  - RACK_ENV=test

runner¶

runner key defines the execution environment of a workflow run.

We currently support two types of environments for running your tasks - Docker) and Linux (Ubuntu) VM.

Docker VM acts as a composable and flexible environment where you can customize it with container images (managed by your organization or community). Whereas, Linux VM is managed by Razorops engineers where most commonly used build tools and programming language runtimes are pre-installed. Both environment types have advantages and disadvantages compared to each other.

To choose Linux VM (Ubuntu) environment, you need to specify os_image property, otherwise we use Docker VM environment by default. An example -

global:
  runner:
    os_image: ubuntu

For Docker-based environment, you define a primary container (the first element in containers property) and other dependencies. All steps in the tasks are executed in the primary container. Razorops also provides CI-optimized container images (on dockerhub) for various languages and frameworks. An example -

global:
  runner:
    containers:
    - image: razorci/ruby:3
    - image: redis

Or if your pipeline needs only a docker image, you can provide a short-hand

global:
  runner: razorci/ruby:3

You can specify os_image and containers both which defines a VM environment to run the steps and the containers running in that environment. This is useful for running various service containers like for databases, caching services, etc. Here is an example -

PostgresRedis

global:
  runner: 
    # use ubuntu VM
    os_image: ubuntu
    # starts postgrses container in background and exposes the ports 
    # locally at localhost:5432
    containers:
    - image: postgres
      environment:
      - POSTGRES_PASSWORD=secret

tasks:  
  test: ## task name
    steps:
    # test connect
    - run: dockerize -wait tcp://localhost:5432
    ...

global:
  runner: 
    # use ubuntu VM
    os_image: ubuntu
    # starts redis container in background and exposes the ports 
    # locally at localhost:6379
    containers:
    - image: redis

tasks:  
  test: ## task name
    steps:
    # Ping test
    - run: (printf "PING\r\n";) | nc localhost 6379

You can specify multiple containers. These will be created/started before executing tasks[].steps and share the host network in the VM (localhost). global.runner property can be overridden by tasks[].runner property if you want to choose different environment for a task.

Note

Currently, we only support container/docker and Linux based environment. The support for Android, Windows, and Macosx based environment is coming soon.

os_image¶

It defines the name of VM image for the runner. Currently we provide ubuntu distribution (jammy/22.04 LTS). It's maintained by Razorops developers and updated at regular intervals. An example -

global:
  runner:
    os_image: ubuntu

containers¶

List of containers that make the environment for running your tasks.

For Docker environment, The first element is called a primary container responsible for executing the steps, while rest are started in the background sharing the host network, generally needed for running tests alongside database, caching or message broker services.

For VM environment, these containers are started in background and will acts as service containers though the steps will run on actual VM. For more details, see "How to use service containers".

Each container must define image property and can optionally have environment, command, and entrypoint properties.

environment defines the environment variables that are injected into the container process. They are defined as key, value pairs separated by = just like global variables and task variables.
command overrides the default CMD when the non-primary container starts.
options are passed as command line options when the non-primary container creates.

global:
  runner:
    containers:
    - image: razorci/ruby:3
      environment:
      - TOKEN=secret
    - image: mysql
      environment:
      - MYSQL_ROOT_PASSWORD=foobar
  ...

Note

command and options keys are not supported for the primary container.

variables¶

It defines environment variables injected into every task of a workflow. They can be interpolated with standard CI variables as well. They are defined as key, value pairs separated by =.

global:
  runner:
    ...
  variables:
  - RACK_ENV=test
  - SHORT_SHA=${CI_COMMIT_SHA:0:8}

tasks¶

A task is the sequence of steps to execute and determines the actions of your pipeline. This is the main object of the configuration with tasks key. Each task has a name, so will sometimes be referred to as a "named task". Tasks will run parallelly they are defined. tasks must have a name that should be unique within the .razorops.yaml file. The relationship between tasks is defined using depends property.

This section defines a list of steps that allow you to build, test and deploy your code. Steps are (by default) executed sequentially, in the order in which they are defined. If a step returns a non-zero exit code, the pipeline immediately aborts and returns a failure status.

Tip

Tasks have a maximum runtime of 1 hour. If your jobs are timing out, consider running some of them in parallel.

Here is an example -

tasks:
  build-image:
    steps:
    ....

  unit-tests:
    steps:
    ...

Each task consists of the tasks's name as a key, list of steps, and many more options. A name should be unique within a current tasks list. The most important key steps, defines list of execution units.

A task allows following parameters -

key	Type	Description
`runner`	map	Defines the build environment for the task. If not provided, `global.runner` will be used instead.
`steps`	array	List of steps with at-least one key which determines it's kind
`variables`	array	Environment variables in variables form
`when`	string	A conditional expression that omits the task when false
`depends`	array	List of tasks that this task depends on. It will only run after the named tasks have completed.

Here is an example of tasks with dependencies:

tasks:
  build:
    ...

  test1:
    depends: [build]
    ...

  test2:
    depends: [test1]
    ...

  deploy:
    depends: [test2]
    ...

The dependencies are defined by setting the depends: key as shown. The deploy: task will not run until the build and test1 and test2 tasks complete successfully. A task must wait until all upstream tasks in the dependency graph have run. So, the deploy task waits for the test2 task, the test2 task waits for the test1 task and the test1 task waits for the build task.

runner¶

It defines the environment for the specific task. Refer here for more details.

It's optional and should be used when you want to choose different build environment for a particular task as it overrides the global.runner property.

os_image¶

Defines the VM image for the specific task. Refer here for more details.

containers¶

List of containers to start for the specific task. Refer here for more details.

steps¶

Steps define an execution plan for a task in a serial order. They are run sequentially till at-least one of the step fails. Razorops has several inbuilt steps that can be included in your pipeline -

Kind	Description
`run`	Execute a command in the shell
`commands`	Execute multiple command(s) in the shell
`checkout`	Clone the source code repository using `git`
`docker/build`	Build and optionally push your Docker image
`docker/push`	Push a docker image into container registry
`cache/pull`	Download objects from the caching storage with given key(s)
`cache/push`	Upload objects to the caching storage with a key
`artifacts/pull`	Download task artifacts
`artifacts/push`	Upload task artifacts
`workspace/persit`	Persist files/directories for downstream tasks (create a layer context)
`workspace/attach`	Restore all layer contexts
`reports/html`	Save HTML coverage reports
`reports/junit`	Save and upload junit files for Insights

You can also specify following properties for a step -

if - defines a conditional expression to execute the step only if evaluated to true.
when - defines a condition to run the step based on the task exit status. It should be one of on_success, on_fail and always (defaults to on_success).
allow_failure - If true (boolean) the step failure won't abort the pipeline.
variables - defines array of environment variables for the step. They are defined as key,value pairs separated by =.
background - If you want to run the step in background and don't care about the exit status of the step. Generally used to start a server process in background without blocking the execution. Refer here for more details.
parallelism - defines a set of tasks with same steps that can be run in parallel. The parallelism property expects integer value larger than 1 (max: 8). For more details, see "Parallelism".
timeout* - defines a timeout in seconds (integer) before the runner kills the step.

Note

Step properties marked as * above, are not supported as of now, will be coming soon.

    steps:
    - checkout
    - docker/build:
        image: example.com/demo
      if: branch == 'master'
      allow_failure: true
    ...
    - run: make clean
      when: always
    ...
    - run: server -p 8080
      background: true
    ...

run¶

This step is designed so you can execute one or more shell commands. The task fails if any of the commands fail or return non-zero exit code. It's an alias for commands step.

tasks:
  unit-tests:
    runner: ruby:2.6
    steps:
    - checkout
    - run: bundle install
    - commands:
        - bundle check
        - bundle exec rspec

commands¶

Similar to run step, it accepts a list of commands to execute in the current environment/shell.

checkout¶

checkout step will perform a git clone of source code repository. Normally this step should be provided at the start of steps in a task which needs all files from the repository. Afterward, you can persist the required files to downstream tasks.

tasks:
  unittest:
    steps:
    - checkout
    - run: echo Hello

docker/build¶

Before deploying anything, you will most of the time want Razorops to build your Docker images. For this you need to use the docker/build task, which is one of the inbuilt step.

The only mandatory parameter of build is the image you want to build:

tasks:
  build-image:
    steps:
    - checkout
    - docker/build:
       image: razorci/docs
       push: true
       tags: ["${CI_COMMIT_SHA}", "latest"]
       dockerfile: Dockerfile
       context: .
       args:
       - API_TOKEN=foobar

docker/build step allows following parameters -

key	Type	Default	Description
`image` *	String		The name of docker image to build
`tags`	Array	`["latest"]`	The list of tags for docker images
`push`	Boolean	`false`	push the image to a docker registry
`args`	Array		Docker build arguments (`--build-args`)
`args-env`	Array		Docker build arguments to be used from secrets configured in Dashboard
`dockerfile`	String	`./Dockerfile`	Path of dockerfile relative to root of your source repo
`context`	String	`.`	Docker context (directory)
`target`	String		target stage in a multistage build (build will run until this stage)

Tip

Always think carefully before pushing your images to a public Docker repository. If you are using a private Git repository or your build contains secrets (e.g. Github token, database passwords) then you need to ensure that you use a private Docker repository. Docker Hub and Quay.io both offer private repositories as part of their subscription plans.

docker/push¶

Push a built image to a remote Docker registry with one or more tags. It supports standard Docker registries, GCR and ECR.

push task supports following parameters:

Key	Type	Default	Description
`image` *	string		Docker image (public or private)
`tags`	array		Multiple tags under which to push the image

Tip

You can optionally push docker image with docker/build task as well with push: true.

cache/push¶

Generates and stores a cache of a file or directory of files such as dependencies or source code in our object storage. Later/downstream jobs can restore this cache.

The cache for a specific key is immutable and cannot be changed once written. For more details, see "Caching dependencies".

Key	Type	Description
`key` *	string	Unique identifier for the cache
`paths` *	array	List of folders or files to cache

Here is an example -

tasks:
  save-npm:
    steps:
    ...
    - cache/push:
        key: npm-cache-{{ checksum "package-lock.json" }}
        paths:
          - node_modules

cache/pull¶

It restores a previously saved cache based on a key. Cache needs to have been saved first for this key like in save-npm step. For more details, see "Caching dependencies".

Key	Type	Description
`key` *	string	Unique identifier of cached object
`fallback_key`	string	Unique identifier of cached object

Here is an example -

tasks:
  restore-npm:
    steps:
    ...
    - cache/pull: 
        key: npm-cache-{{ checksum "package-lock.json" }}
        fallback_key: npm-cache-

artifacts/push¶

Generates and stores an artifact of a file or directory of files (for example, distribution zip files or jar files) in our object storage so that they can be downloaded later.

The cache for a specific key is immutable and cannot be changed once written.

Key	Type	Description
`name` *	string	Artiface name
`paths` *	array	List of folders or files to include

Here is an example -

tasks:
  save-npm:
    steps:
    ...
    - artifact/push:
        name: binaries
        paths:
          - bin

artifacts/pull¶

It downloads a previously saved artifact within the pipeline or different pipeline in current workspace. Artifact needs to have been pushed first.

Key	Type	Description
`name` *	string	Unique identifier of artifact
`pipeline`	string	Pipeline identifier

Here is an example -

tasks:
  restore-npm:
    steps:
    ...
    - artifacts/pull:
        name: binaries

workspace/persist¶

Persist a collection of files or directories such as build jars or compiled artifacts for downstream tasks. For more details, see "Working with layers".

The cache for a specific key is immutable and cannot be changed once written.

Key	Type	Description
`paths` *	array	List of folders or files to persist

Here is an example -

tasks:
  save-npm:
    steps:
    ...
    - workspace/persist:
        paths: [node_modules]

workspace/attach¶

It downloads saved workspace layers from upstream tasks within a workflow run. Layers needs to have been pushed first. For more details, see "Working with layers".

Here is an example -

tasks:
  restore-npm:
    steps:
    ...
    - workspace/attach

reports/html¶

If you are running unit/integration tests which generate coverage report or any html files which you need to introspect after the completion of a workflow, you can save them using this step.

Here is an example -

tasks:
  restore-npm:
    steps:
    ...
    - reports/html:
        dir: coverage
        index: index.html

reports/junit¶

You can configure a step to use JUnit test reports, and Razorops will display a report in Insight section in the piepline in dashboard. It helps to identify the most failing and slowest test without having to check the entire log.

Here is an example -

tasks:
  restore-npm:
    steps:
    ...
    - reports/junit:
      - spec/*.xml
    ...
    # or
    - reports/junit:
        paths: [spec/1.xml, spec/2.xml]

when¶

You can filter the execution of each task by using the when key in the YAML file. This can be done with an expression, as you can see in the following sample configuration:

tasks:
  build-image:
    steps:
    ...
    when: branch == 'master'

The expression has access to the workflow related context objects branch, tag, and event. Read here for the more details.

depends¶

Tasks can be define in a direct acyclic graph to build relationships between them and finish in quickest manner possible. Such dependencies are defined between tasks using depends property. This lets you run some tasks without waiting for other ones.

Here is an example

tasks:
  a:
    steps:
      ...

  b:
    depends: [a]

  c:
    depends: [a]

In this example, the task b and c will be executed in parallel as soon as a completes.

trigger¶

By default, Razorops will create a new workflow on every code change to the your repository. You can limit the branches or tags that you want to run with trigger property.

when¶

It helps to combine if and unless property. You can also define a shorthand to infer the if expression.

if¶

Razorops will trigger a new workflow run if result of if statement is evaluated to be true. For more details, see "Conditional expressions".

trigger:
  ## triggers pipeline only on feature branches and on tags (mind the prefix)
  when:
    if: branch =~ '^feature-*' || tag =~ '^v.*'

Or you can define a shorthand -

trigger:
  ## triggers pipeline only on feature branches and on tags (mind the prefix)
  when: branch =~ '^feature-*' || tag =~ '^v.*'

unless¶

Razorops will not trigger a new workflow run if result of unless statement is evaluated to be true. For more details, see "Conditional expressions".

trigger:
  ### triggers pipeline on non-master branches
  when:
    unless: branch == 'master'

schedules¶

It allows you to trigger a new workflow at a scheduled time periodically. You can specify multiple schedules with different frequency. For example -

trigger:
  schedules:
  - name: daily
    cron: "15 12 * * *"
    branch: master

Each item in schedules key supports the following properties -

name - defines an unique name for the schedule in the pipeline
branch- defines the branch to run the new workflow against.
cron - It uses cron patterns to define the shcedule to repeat. You can use the crontab.guru tool to check that a CRON pattern matches the intended schedule.

For more information on schedules, please follow Scheduled workflows section.

Note

Scheduled workflows run on the default or provided branch. The shortest interval you can run scheduled workflows is once every 5 minutes.

workflows¶

workflows defines multiple possible execution paths (workflow runs) with their run order of tasks (defined under tasks property) to model complex pipelines. For more details, check "Workflows and Tasks". Here is an example:

tasks:
  ## define build, test, publish and deploy tasks
  ...

workflows:
- name: test-build
  tasks:
  - build
  - test
  - publish:
      depends: [build, test]

- name: test-build-deploy
  when: branch == "master"
  tasks:
  - build
  - test
  - publish:
      depends: [build, test]
  - deploy:
      depends: [publish]

Each element in workflows key supports following properties:

name - defines an unique name for the workflow run.
tasks - It defines the run order of the tasks in the workflow run. Each element in this array should be defined under tasks section (refer line 1 in example). Checkout tasks to know more about how to define a task in Pipeline YAML.
tasks[].depends - List of tasks which should be completed before this task. The dependencies should be included in the array (refer line 8,9 in example) and defined under tasks (refer line 1 in example)
when - You can skip the execution of the workflow by using the when key. This can be done with a conditional expression. For more details, check "Conditional Expressions".

workflows key is optional and only needed to build a complex CI process. Alternatively, you can also define a DAG of tasks using depends key under tasks.