Chains

A job's config/chains file defines its pipeline. It ranges from a single image to a sequence of steps that pass output files to the next step. You write it to the job's config/chains path:

report_builder

Referencing a repo image

A step points at a repo image. There are two reference forms:

• Short form — a bare repo name resolves to your latest image: report_builder → @<your_username>/report_builder:latest. Add a label to pin one, as in report_builder:prod.
• Canonical form — name the owner and label explicitly: @novem_demo/report_builder:latest. Use this to reference an image shared with you by another user.

The label is one of the repo's image labels: latest for the default branch, prod for the newest tag (the default branch until the repo has one), or tag:<name> for one tag, as in @novem_demo/report_builder:tag:v1.0.0. Branch names are not labels, and a step naming a label the repo does not have fails the run.

A label is resolved once, when the run is submitted. The run records the exact image each step resolved to and the commit it was built from, for the duration of the run. A new build of latest does not change an existing run; subsequent runs use the new image. The run log names what each step resolved to.

Referencing another job

A step can run another novem job instead of a repo image. Reference it by path: /j/<job> for one of your own jobs, or /u/<user>/j/<job> for a job shared with you by someone else.

fetch   <- data_fetcher
rollup  <- /j/nightly_rollup

fetch -> rollup

At run time novem expands the referenced job into the steps it is built from and splices them into the chain, so rollup runs whatever nightly_rollup runs, fed by fetch's output. A referenced job may itself reference further jobs; novem expands the whole tree, rejecting reference cycles and nesting that runs too deep. Each referenced job records its own run, nested under this run — see runs.

Each referenced job runs with its own environment variables; a referenced job's steps do not inherit the environment of the job that composes them.

Permissions. You must have execute access to every job your chain references directly — your own jobs, or another user's job shared with you with execute permission. You do not need separate access to those jobs' internal dependencies. Likewise, a user with execute access to your job does not need separate access to the jobs it references.

A single step

The simplest chain is one reference. This runs that image and is exactly what the quick start uses.

report_builder

Multi-step pipelines

To pipe steps together, use the chain grammar. It has two kinds of line:

• Bindings assign a short label to a repo image with <-.
• Edges wire labels together with ->, left to right.

fetch  <- data_fetcher
render <- chart_builder

fetch -> render

This runs fetch first, then pipes its output files into render as that step's input. The labels keep the pipeline readable and let you reference the same image more than once.

Edges can be chained in a single line for longer pipelines:

fetch     <- data_fetcher
transform <- transformer
render    <- chart_builder

fetch -> transform -> render

Each step receives the previous step's output as its input; the final step's output becomes the job's result.

Fan-out

Prefix a step with a dot to fan out over the previous step's output: the step runs once per output file, in parallel. A fan-out step sits between a producer and a barrier — the step before it supplies the files to spread over, and the step after it waits for every parallel instance and collects their output — so a fan-out is never the first or last step in a chain.

split   <- splitter
render  <- chart_builder
collect <- collector

split -> .render -> collect

Here render runs once for each file split produced, and collect receives all of their outputs together once the instances finish. Fan-out currently supports the per-file mode shown above.

Fan-out steps can follow one another. In split -> .extract -> .render -> collect, split's files are spread over extract, and each render instance runs on the output of the one extract instance that came before it, so every file goes through both steps in its own instance. collect gathers what all of the render instances produced.

A chain's fan-out steps must form one consecutive run, with a predecessor to spread over and a successor to collect into. Two fan-outs with an ordinary step between them, or a fan-out at either end of the chain, is a configuration error.

Holds

Put a ! on a step to stop the run there and wait for a person. The mark is positional and goes in an edge line, next to the step's name:

!a -> b     stop before a runs
a! -> b     stop after a has run
!a! -> b    both: look at a's input, then at its output

A run that reaches a hold parks the files at that point in a space, reports waiting, and continues only when someone with execute access on the job answers. Whatever is in the space then becomes the next step's input, so a reviewer can correct, add or delete files before the run goes on. See runs waiting at a hold for the review itself.

Holds need spaces, which are gated behind a per-account feature flag. On an account without them, a chain with a hold fails when it is triggered, and the run log says why. A reviewer needs spaces too, to open the parked files.

A space cannot hold every file name. It refuses < > : " | ? * \, control characters, a name ending in a dot or a space, and device names such as CON or NUL. A name may be at most 255 bytes, and a path at most 1024 bytes and 64 folders deep. It also compares names without case, so Report.csv and report.csv would be one file. If a hold would park such a file, the run fails before parking anything, and the log names the file.

extract <- data_extractor
render  <- report_builder

extract! -> render

Here render runs only after someone has looked at what extract produced.

A step that exists only to stop binds to _ instead of an image:

extract <- data_extractor
audit   <- _
render  <- report_builder

extract -> !audit -> render

A _ step runs nothing, so it must carry a hold, and either mark parks before it: audit! is the same as !audit. A _ step with no mark is a configuration error. A hold on the first step parks the trigger's own input, which makes a leading ! a manual start for a scheduled job. A hold after the last step (render!) is a sign-off: nothing runs after it, so the files the reviewer leaves in the space become the run's output.

A chain may hold at several steps, and each one waits for its own answer in turn. A hold cannot be a fan-out step: .!b and .b! are configuration errors. Nor can a hold come right before a fan-out, as in split! -> .render or a _ stop in that place, because the run resumes at the fan-out with no step's output to spread over. Put a step between them to pass the files on. Because the mark lives in an edge line, a chain needs at least one -> line to carry a hold.

Step resources

A step defaults to one CPU, 200 MB of memory, and a 1 GB disk. To request different resources, add a comma-separated list after the image reference:

extract  <- data_extractor
crunch   <- number_cruncher,cpu=4,mem=4G
report   <- report_builder,timeout=30s

extract -> crunch -> report

Four are available:

The single-reference form takes them too: report_builder,mem=2G.

timeout can shorten a step's existing time limit. For example, set a shorter limit for work with a known expected duration. A value above the existing limit is a configuration error.

What your plan allows

Resources are capped per step, by plan. Asking for more than your plan allows fails the run with a message identifying the step and the limit. Requested resources are not reduced automatically.

Free jobs run at the defaults and cannot set cpu, mem or disk — a chain that sets one fails rather than ignoring it. timeout is not a plan limit, so any job may shorten a step.

The ceiling is per step. On a fan-out step it is therefore per parallel instance: mem=4G on a step that spreads over 20 files means each of those instances gets 4G, not 4G between them. The worker's available slots determine how many instances run concurrently. Account for each instance when estimating the step's total resource use.

Shape rules

A chain is a single linear sequence of steps. Each step has at most one step before it and one after it. Once the file declares at least one -> line, every bound step must be connected to the chain — a step nothing points at is an error rather than a step that never runs. A file with no -> line at all is not validated for connectivity: its bound steps run in definition order.

Two shapes are therefore not available. Branching, where one step feeds two others:

A -> B
A -> C

And merging, where one step takes input from two others:

A -> C
B -> C

Run several independent things by fanning out over files instead, or by splitting the work across separate jobs.

There is also no comment syntax. Every non-blank line is either a binding or an edge, so a line that is neither is a configuration error.

How it's stored

novem normalises whatever you write into a canonical form — bindings first, then the edge line — so reading config/chains back may look slightly different from what you wrote:

fetch <- @you/data_fetcher:latest
render <- @you/chart_builder:latest

fetch -> render

Next steps

• Jobs overview — env vars, schedules and running.
• Repos — build the images your chain references.