Caching dependencies¶

In Razorops, every workflow starts with a clean slate in a virtual machine and reaped after it finishes. Often you might be downloading dependencies from the public internet required for running tests or compiling the source code causing longer workflow runs and wasting network bandwidth. Luckily you can cache these frequently used dependencies. It's an effective way to make tasks faster.

Tip

To make your workflows faster and more efficient, you can create and use caches for dependencies and other commonly reused files.

Example¶

You can use cache/pull step to restore dependencies with an unique key and a fallback key. In case of cache miss, the fallback key will be used and cache/push step to upload the dependencies from paths.

Here is an example to cache bundler dependencies.

    steps:
    ...
    - cache/pull:
        key: bundle-{{ checksum "Gemfile.lock" }}
        fallback_key: bundle-
    ...

    - cache/push:
        key: bundle-{{ checksum "Gemfile.lock" }}
        paths:
        - vendor/bundle

How to use¶

Use cache/pull, cache/push steps to cache data that makes your workflows faster. In the case of a cache miss or zero cache pull, it will fail silently and won't break your running pipeline. The best example would be to use this with your package managers such as Yarn, Mix, Bundler or Maven. After your initial download, you can build a cache and then you can pull that cache in your next workflow run.

With pulled dependencies from a cache, commands like mix, bundle or yarn will only need to download new dependencies, rather than re-download every package on each and every run.

Cache key¶

Cache key template syntax is very basic. You just need to provide a string. In that string you can use variables by prefixing them with a . in {{ }} construct, from provided metadata object (see below). Each key is namespaced to the pipeline.

Also following helper functions provided for your use:

Function	Description
`checksum`	Provides md5 hash of a file for given path
`epoch`	Provides Unix epoch
`arch`	Provides Architecture of running system
`os`	Provides Operation system of running system

Metadata

Key	Description
`Repo`	Pipeline name
`Number`	Workflow run identifier
`Branch`	Commit branch
`Revision`	Commit revision
`Env.<name>`	Access Pipline or standard CI environment variables

Template Examples

    "{{ .Repo }}-{{ .Env.CI_REPO_BRANCH }}-{{ checksum "go.mod" }}-yadayadayada"

    "{{ .Repo }}_{{ checksum "go.mod" }}_{{ checksum "go.sum" }}_{{ arch }}_{{ os }}"

Pull cache¶

cache/pull step can be used to pull data with a cache key. You can optionally specify a fallback key which will be used in case of cache miss.

Push cache¶

cache/push step can be used to upload dependencies with a cache key. The directories and files will be archived and uploaded to our storage, which can be pulled in future runs. It will also override the data against the key if exists in storage.

cache/push supports following keys -

key - a cache key template.
paths- an array of directories or files to cache. Each path should either be relative to the current workspace or absolute path. It doesn't support glob patterns.

Usage limits¶

Razorops will keep a cache maximum upto 15 days. We recommend to keep the maximum size of a cache less than 500MB because larger the size longer the runtime and increased network utilization. It's better to split larger cache sizes into multiple distinct caches to minimize flaky behaviour during download.

Examples¶

Nodejs¶

Assuming your project is using npm or yarn to install the Node.js dependencies.

yarnnpm

steps:
    ...
    - cache/pull:
        key: yarn-cache-{{ checksum "yarn.lock" }}
        fallback_key: yarn-cache-{{ .Branch }}
    - run: yarn ci
    - cache/push:
        key: yarn-cache-{{ checksum "yarn.lock" }}
        paths:
        - ~/.cache/yarn
    ...

steps:
    ...
    - cache/pull:
        key: npm-cache-{{ checksum "package-lock.json" }}
        fallback_key: npm-cache-{{ .Branch }}
    - run: npm ci
    - cache/push:
        key: npm-cache-{{ checksum "package-lock.json" }}
        paths:
        - node_modules
    ...

PHP¶

The following example will cache the Composer dependencies -

steps:
    ...
    - cache/pull:
        key: composer-{{ checksum "composer.lock" }}
        fallback_key: composer-{{ .Branch }}
    - run: composer install -n --prefer-dist
    - cache/push:
        key: composer-{{ checksum "composer.lock" }}
        paths:
        - vendor
    ...

Ruby¶

The following example will cache the Bundler dependencies -

steps:
    ...
    - cache/pull:
        key: bundle-{{ checksum "Gemfile.lock" }}
        fallback_key: bundle-{{ .Branch }}
    - commands:
      - bundle config set --local path 'vendor/bundle'
      - bundle install
    - cache/push:
        key: bundle-{{ checksum "Gemfile.lock" }}
        paths:
        - vendor/bundle
    ...

Python¶

If you are using Pip, Pipenv or virtualenv to manage your dependencies -

PipPipenv

steps:
    ...
    - cache/pull:
      - key: pip-cache-{{ checksum "requirements.txt" }}
    ...
    - run: pip install -r requirements.txt --user
    ...
    - cache/push:
      - key: pip-cache-{{ checksum "requirements.txt" }}
        paths:
        - ~/.cache/pip ## for pip

steps:
    ...
    - cache/pull:
        key: pip-packages-{{ checksum "Pipfile.lock" }}
    - commands: 
        - pip install pipenv
        - pipenv install --dev --deploy
    - cache/push:
        key: pip-packages-{{ checksum "Pipfile.lock" }}
        paths:
        - ~/.local/share/virtualenvs
    ...

Java¶

If you are using Maven or Gradle to manage your dependencies -

mvngradle

steps:
    ...
    - cache/pull:
        key: mvn-cache-{{ checksum "pom.xml" }}
    - run: mvn -q dependency:go-offline test-compile
    - cache/push:
        key: mvn-cache-{{ checksum "pom.xml" }}
        paths:
        - ~/.m2
    ...

steps:
    ...
     - cache/pull:
        key: gradle-cache-{{ checksum "build.gradle" }}
        fallback_key: gradle-repo-{{ .Branch }}

    - run: gradlew assemble
    ...

    - cache/push:
        key: gradle-cache-{{ checksum "build.gradle" }}
        paths:
        - ~/.gradle
    ...