Pipelines
Pipelines build, test and package a project when changes land, on demand, on a schedule, or after another pipeline. They're YAML files in the repo, so they're versioned and reviewed like everything else, and they run on your own machines.
Files
Put one file per pipeline in .lodebase/pipelines/ (.yml or .yaml). Each branch uses its own copy. .lodebase/** is owned by maintainers by default (an implicit .lodeowners line), since pipelines decide what required checks mean: contributors change pipelines through merge requests that a maintainer approves.
yaml
name: Build # default: the file name
on:
submit: # a changeset lands
branches: [main, "release/*"] # default: every branch
paths: ["Source/**", "*.uproject"]
paths-ignore: ["Docs/**"]
manual: # `lode pipeline run`, or the web page
inputs:
config: { default: Development, options: [Development, Shipping] }
notes: { description: "Shown in the log", required: false }
schedule:
- cron: "0 3 * * *" # UTC; a leading seconds field is allowed
branch: main # default: the repo's default branch
run_completed: # after another pipeline finishes
pipelines: [Build]
states: [success] # default: any outcome
branches: [main]
merge_request: # a merge request into these branches, before it lands
branches: [main]
paths: ["Source/**"]
env:
UE_ROOT: /opt/UnrealEngine
CONFIG: ${{ inputs.config }}
jobs:
editor:
name: Editor (${{ inputs.config }})
runs-on: [linux, ue5] # a runner needs every label
timeout-minutes: 120 # default 360
workspace:
clean: false # wipe the workspace first
view: ["Source/**", "Config/**", "*.uproject"] # only sync these
env: { TARGET: GameEditor }
steps:
- name: Compile
run: $UE_ROOT/Engine/Build/BatchFiles/Linux/Build.sh $TARGET Linux $CONFIG -project="$PWD/Game.uproject"
- uses: lodebase/upload-artifact@v1
with: { name: editor, path: "Binaries/**" }
test:
needs: editor
runs-on: [linux, ue5]
steps:
- uses: lodebase/download-artifact@v1
with: { name: editor }
- run: ./RunTests.sh
- name: Report
if: failure()
run: echo "tests failed on ${{ branch }} CL ${{ number }}"
package:
needs: [editor, test]
if: branch == 'release/*'
runs-on: [windows]
steps:
- shell: pwsh
run: .\Package.ps1 -Token "${{ secrets.STEAM_TOKEN }}"Jobs and steps
needsmakes a graph; jobs without it start right away, in parallel if there are runners for them.- A job runs when everything it needs succeeded.
if:changes that:always(),failure(),cancelled(),success(), comparisons (branch == 'release/*'globs),&&,||,!. - Steps run in order; after a failure the rest are skipped unless their
if:says otherwise (always()for clean-up).continue-on-error: truelets a step or a job fail without failing the run. shell:isbash(default on Linux/macOS, with-eo pipefail),sh,pwsh(default on Windows),powershell,cmdorpython.working-directory:is relative to the workspace.
Expressions
${{ … }} works in run, with, env, name and working-directory:
| Name | Value |
|---|---|
repo, branch, number | where the run builds (the changeset number) |
pipeline, trigger, actor | the pipeline's name, what started it, who |
run.number, run.id | the run's number in the repo (#12) and its id |
inputs.<name> | manual inputs (with defaults filled in) |
env.<NAME> | pipeline and job env |
needs.<job>.result | success, failure, cancelled or skipped |
secrets.<NAME> | a repo secret (see below) |
Steps also get CI=true, LODEBASE_REPO, LODEBASE_BRANCH, LODEBASE_NUMBER, LODEBASE_PIPELINE, LODEBASE_RUN_NUMBER, LODEBASE_JOB, LODEBASE_WORKSPACE and LODEBASE_RUNNER in their environment.
Runners
A runner is lode-runner on a build machine. It takes one job at a time.
sh
lode runner token # a maintainer, in a workspace of the repo
lode-runner register --server https://app.lodebase.dev --token lbreg_... --name build-01 --labels ue5,gpu
lode-runner run # or as a service (below)- The token is single-use and expires after an hour.
lode runner token --global(server admins) registers a runner for every repo. - The OS label (
linux,windows,macos) is added automatically. - Each repo gets one persistent workspace under the work directory. Before a job, the runner syncs it to the job's changeset incrementally: only files that changed since its last job (or that a build step modified) are written, and build outputs that aren't in the repo stay, so incremental compiles work.
workspace: { clean: true }starts from scratch. - Downloaded content goes into the runner's own chunk cache, so a 40 GB project is downloaded once.
- Cancelling a run stops the step's whole process tree. A runner that restarts mid-job reports that job as failed; one that disappears is failed after three minutes without a heartbeat. Jobs past their timeout are stopped.
As a systemd service:
ini
# /etc/systemd/system/lode-runner.service
[Service]
User=build
ExecStart=/usr/local/bin/lode-runner run
Restart=always
[Install]
WantedBy=multi-user.targetOn Windows: sc.exe create lode-runner binPath= "C:\LodeBase\lode-runner.exe run" start= auto under a build account (or run it in a logged-in session if your builds need a desktop).
Shared runners and autoscaling
Repos that don't bring a runner can use the server owner's machines, within limits:
- Monthly minutes per repo (
LODEBASE_SHARED_MINUTES, default 500). Each job counts from start to finish, rounded up, at least one minute. The month is the calendar month in UTC. Admins change a repo's allowance withlode runner usage --set 1000(or--default). - A cap per job (
LODEBASE_SHARED_JOB_MINUTES, default 60): a job on a shared runner is stopped then, whatever its owntimeout-minutessays. - Own runners first. If a repo has a runner of its own (seen in the last ten minutes) with the labels a job needs, that job never goes to a shared runner and uses no minutes.
- Out of minutes: jobs that only shared runners could take fail at once, with a message saying how to add a runner, instead of waiting.
lode runner usage and the Pipelines page show where a repo stands.
Pools: environments on demand
Shared capacity comes from pools described in a YAML file (LODEBASE_POOLS_FILE, see deploy/pools.example.yml). For each waiting job the server starts a fresh container, the runner in it takes that one job, and the container is removed when the job ends. Nothing runs while nothing is queued.
yaml
limits: { linux: 4, windows: 1 } # most environments per OS, across pools
pools:
- name: linux
os: linux
max: 4
warm: 0 # keep some started and waiting, for slow starts
provider: { type: docker, image: lodebase/runner:latest, host: "ssh://ci@build-box", memory: 8g }- A job goes to the first pool that has all of its
runs-onlabels. Every pool has its OS name andsharedas labels, plus the ones you list. - Each container gets a one-time credential through its environment (never on a command line). It stops working once the runner has claimed its job, and the runner is deleted with the container. Jobs run as an unprivileged user in a workspace that starts empty every time.
- An environment that doesn't check in within
boot_timeout_secs(default 300, 900 for Windows) is removed. So is one that stops before taking a job. Containers the server doesn't know about (after a crash) are removed within a minute. - If starting a container fails, the pool waits before trying again (15 s, doubling up to 10 min);
lode runner poolsshows the last error. type: dockerworks with Docker or Podman (command: podman). Usehost(orDOCKER_HOST) to run builds on a separate machine over SSH rather than giving the server the host's Docker socket, and give the containers a network that can reach the server but not your LAN.- Windows: a Windows machine running Docker with Windows containers can be a pool of its own (
os: windows,host:pointing at it).deploy/runner/Dockerfile.windowsis a starting point; this path isn't tested yet.
Images: deploy/runner/Dockerfile builds a Debian image with lode-runner, bash, git, python3 and build tools. Build your own FROM it with your SDKs.
Download cache per repo. A pool environment is started for one repo's waiting job. It only takes that repo's jobs and mounts that repo's cache (lodebase-cache-<pool>-repo<id>, a Docker volume), so the second build of a 40 GB project downloads only what changed, while the workspace still starts empty. The runner trims the cache to LODEBASE_CACHE_MAX_GB (default 50) after each job. cache: false on a pool turns this off; warm spares have no cache (they take any repo). Old volumes of deleted repos aren't removed yet: docker volume ls | grep lodebase-cache.
lode runner pools (server admins) lists pools, running environments and recent errors. lode runner token --shared registers a long-lived shared runner by hand; unlike pool containers it keeps its workspace between jobs, so only use it on machines where that's acceptable.
Merge requests and required checks
A pipeline with a merge_request trigger runs whenever a merge request into a matching branch is opened or gets a new revision. It builds what would land: the target branch's current head with the merge request's changes on top. A new revision cancels the previous revision's runs.
Branch protection can require checks by pipeline name:
sh
lode protect set main --approvals 1 --check Build --check Tests
lode mr checks 12 # the checks on the latest revision
lode mr checks 12 --rerun # run them again (or --rerun Build)A required check blocks merging until its latest run on the latest revision passed. If no pipeline produced it (for example a paths filter didn't match), the merge request says so; --rerun Build runs it regardless of paths. Admins can still merge past failing checks with --bypass "reason" (audited). Auto-merge waits for the checks.
Merge queue
sh
lode protect set main --check Build --merge-queue
lode mr merge 12 # queues it
lode mr queue # what's waiting, what's being tested
lode mr dequeue 12Checks on a merge request ran against the branch as it was then. With the merge queue, "merge" instead puts an approved, green merge request in line. The queue takes entries one at a time, runs the required checks again on the branch's latest head plus the change (trigger merge_queue), and lands it only if they pass. Two changes that each passed on their own can't break the branch together: the second is tested on top of the first, and leaves the queue if that fails. Entries leave the queue if they get a new revision, are closed, or fail; the branch moving under a test (an admin bypass) restarts that test. Entries are tested one after another, not in speculative batches yet.
Secrets
sh
lode secret set STEAM_TOKEN # reads the value from stdin
lode secret list
lode secret rm STEAM_TOKENMaintainers manage secrets (also in the web app, under Pipelines → Secrets and runners). Values:
- are encrypted with a per-secret key, itself wrapped by the server's master key (
LODEBASE_SECRETS_KEY, 32 bytes:openssl rand -base64 32), which never goes in the database. Back it up: without it the secrets can't be read. - are write-only: no API returns them.
- go only to the jobs that refer to them (
${{ secrets.NAME }}), only when the job starts, and each release is in the audit log. - are masked (
***) in logs. - can be protected (
lode secret set NAME --protected, or the checkbox in the web app): then they only go to runs on protected branches, never to merge request runs or runs on task branches.
Anyone who can open a merge request can change what its pipelines run (the pipeline files are read from what would land), so treat unprotected secrets as readable by everyone with write access. Release and signing keys belong in protected secrets.
Manual inputs can't smuggle in ${{ … }}: the server neutralises them.
Job tokens
A runner's own credential can only claim jobs. Each job gets a token, valid only while it runs, that can read its repo and store artifacts. It can't submit, lock, review or reach other repos, and steps never see it.
Using it
sh
lode pipeline list # files on your branch, with errors
lode pipeline run Build -i config=Shipping --watch
lode pipeline runs
lode pipeline show 12
lode pipeline log 12 [job] --follow
lode pipeline cancel 12
lode runner listThe web app's Pipelines tab lists runs and pipelines, starts manual runs, follows logs live, and lists artifacts with download links.
Downloading artifacts
- Browser: a run's page has Download all (.zip), and a .zip link per artifact. The archive is streamed straight from storage (no waiting while the server builds it) with its exact size up front, so the browser shows real progress. Files are stored uncompressed: builds are mostly compressed already, and it keeps big downloads fast. Archives over 4 GB use ZIP64, which Windows Explorer, macOS, 7-Zip and
unzipall open. - Terminal:
lode pipeline download 12writes every artifact of run #12 to./artifacts/run-12/<artifact>/...(-a gamefor one,-o DIRfor elsewhere). It goes through your daemon's chunk cache, so anything you already have (the previous nightly, files that didn't change) isn't downloaded again; executable bits are kept.
Retention removes old runs, logs and artifacts: see retention-and-gc.md. lode pipeline keep 12 keeps a run you'll want later (a release build).
The tray tells you when a run you started (or your submit started) passes or fails.
Not yet
- The merge queue tests one entry at a time (no speculative batches).
- One job per runner process; run several runners for more.
- Self-hosted runners run steps directly on their machine; only pool containers are isolated.
- Pools start containers only (Docker/Podman). VMs are a later option.