Skip to content

Running your builds on your own machines

A native Groves machine of your team can take ordinary pipeline work: container steps and image builds. Three things have to agree before a step goes there, and each can say no:

  1. The team’s execution policy says which work may leave Canopy’s machines.
  2. The owner of the machine says which work it takes, when it joined: canopy forest join --accept steps,image-builds,secrets. Nothing is accepted by default.
  3. The step itself: a step that names native: still runs where it names; a pull request from a fork never leaves Canopy’s machines.
canopy execution-policy show --team <team_id>
canopy execution-policy set --team <team_id> --own-machines allowed --image-builds allowed
canopy execution-policy show --env <environment_id> --effective

The team’s policy is the ceiling: a project or an environment can only restrict it. own_machines and image_builds are off, allowed or preferred (the default is off); fallback says whether work may go back to Canopy’s machines when none of yours can take it, and step_secrets and open_egress default to the safe side. A step can also ask in the pipeline:

step('test', { image: CI_IMAGE, run: 'bun test', placement: { providers: ['own', 'canopy'] } });

A step can also name one machine or a pool of your machines, and say what to do when none can take it:

step('test', { image: CI_IMAGE, run: 'bun test', placement: { providers: ['own'], grove: 'my-mac', unavailable: 'fail' } });

The step’s own words come first, and the team’s policy is still the ceiling:

  • providers limits where the step may run. A step that leaves canopy out never goes to Canopy’s machines, whatever the team’s fallback says.
  • grove and pool limit the machines of yours that may take it. A step that names a machine is never moved to another one or to Canopy’s machines.
  • unavailable: 'fail' fails the step at once when no allowed machine can take it, with the reason each machine gave. The default, wait, queues it until the wait ends.
  • The team’s fallback lets a step with no such restriction go back to Canopy’s machines. It never overrides a step’s own restriction.

canopy pipelines plan <graph> --explain --verbose --node <node> prints where a step could run and why not on each machine, for ordinary steps and image builds the policy offers to your machines as well as for steps that name one:

Placement now:
build-image (offered to your own machines first)
ready: singularity-9dcc
refused: gios-macbook-pro-9d5a - incompatible:platform_mismatch (this machine builds linux/arm64, the image is for linux/amd64)

A machine is ready when it could take the step now, waiting when it will once something changes (asleep, paused, full), and refused when it cannot, with the reason. When more than one machine is ready, the one with the most free budget is chosen. A step that is not offered to your machines (Canopy compute comes first in your policy) shows only the policy.

  • Container steps run in a read-only container on a Linux machine, with the CPU share low enough that your own work on the machine wins, and the network off when the step declares none. A step that lists hosts, and a step that uses secrets on a machine whose owner has not allowed secrets, stay on Canopy’s machines.
  • Container steps on a Mac run in a Linux guest created for that one step, with your step’s directories shared into it and no other part of the Mac visible. The guest has no network interface, gets 2 CPUs and 2 GiB unless the step names limits, and is removed when the step ends. A Mac takes a linux/arm64 step, and a linux/amd64 step under Rosetta, only after its agent ran a test step in a guest of that platform. A step that lists hosts in egress runs there too: the guest reaches only those hosts, on ports 80 and 443 and never a private address, through a proxy on the Mac that the guest talks to over a relayed socket, and the Mac takes such steps only after a guest proved that path (the allowed host answers, another is refused, going round the proxy fails). A step that needs the open network stays on another machine. A step that names native: is not affected.
  • Image builds (Dockerfile and buildpacks) run in a rootless BuildKit container on a Linux machine, and in a Linux guest created for that one build on a Mac. The image records which machine built it.
  • Several platforms: canopy environments update <environment_id> --build-platforms linux/amd64,linux/arm64 builds one image per platform and publishes one image. Canopy has no arm64 build machine, so the arm64 image needs a machine of yours; the build plan fails before anything runs when there is none. See Native Groves.

A step on your own machine is not sandboxed from that machine the way a step on a Canopy build machine is.

  • A capability is offered only while it works. A machine offers image builds only after a real test build passed on it, and it runs that test again every six hours: if Docker has stopped, the builder image is gone or the login no longer works, the machine stops offering image builds at once, says why in its log, and offers them again when the test passes.
  • A job cannot fill your disk. Every 15 seconds the agent reads the free space of the disk the jobs live on, and a job is stopped when it falls below 2 GiB or 5% of the disk, whichever is larger. The job fails with a reason that names both numbers, and the machine keeps what it needs. The Docker data directory is not watched.
  • A container step or image build runs with the limits above, but the network is open unless the step has none or lists the hosts it needs, and a Dockerfile can reach everything the machine can reach, including other machines on your network. A container step that lists hosts in egress runs on a Linux machine in a container with no network at all: it reaches only a proxy the agent runs, through a unix socket, and the proxy lets through only those hosts, on ports 80 and 443, and never a private address. It cannot reach the machine’s own services or any other address, and the proxy of one job is not reachable from another. An image build still has an open network.
  • A Mac builds and runs container steps in a guest, but a step that names native: runs as an ordinary process of the user the agent runs as.
  • Whatever a step reads from your repository is on the machine while it runs.
  • Your own work and a step share the machine’s memory and disk; the owner’s controls (pause, hours, a battery stop, a CPU and memory budget) say how much a step may use.

Use machines you would trust with the code of the repositories whose steps you send there.

canopy forest config
canopy forest config set --paused true
canopy forest config set --window 22:00-07:00 --refuse-on-battery true --cpu-budget-milli 4000 --memory-budget-mib 8192
canopy forest config set --refuse-on-low-power true --refuse-when-hot true

canopy forest config runs on the machine itself and changes only the flags you pass. On a Mac it asks for your password, because the agent’s settings belong to its service user. canopy forest status shows each machine’s budget, what running jobs use and, when it is not accepting jobs, why.

A paused machine, one outside its hours, one on battery, one in Low Power Mode or one slowed down by heat (when you ask for those) stops accepting work; a step meant for it waits (owner_paused) or goes to another machine. canopy forest groves update <name> --portable-steps on --image-builds on --secrets on changes what a joined machine accepts.

Each job holds a lease the machine renews every time it reports in. A machine that stops reporting loses its jobs after the lease runs out (five minutes): a container step that can safely run again is run again on another machine and is not counted against its retries, and anything else fails with a message that the machine was lost. A step that ran on a machine of yours is reused later only while your policy still lets that step run there.

canopy builds show <build_id> --env <environment_id> and the pipeline view name the machine a step ran on and, for a multi-platform build, the machine that built each platform. In the app, a step that was offloaded to one of your machines has a small laptop icon next to its name on the build page, and hovering it names the machine; a step that ran on Canopy’s compute has none. The Performance page of a machine is under Native Groves.

Building linux/amd64 images on an Apple silicon Mac

Section titled “Building linux/amd64 images on an Apple silicon Mac”

A Mac builds linux/arm64 natively. It can also build linux/amd64 under Rosetta, and offers that only after it built a test image for linux/amd64 in a guest itself (the agent’s own preflight, no flag to set; the machine’s owner can switch it off with FOREST_IMAGE_BUILD_EMULATED_AMD64=false). The team’s execution policy decides whether a build may use emulation: canopy execution-policy set --team <team_id> --emulated-builds allowed (the command replaces the whole level, so repeat the other settings you want to keep). An emulated build is slower than a native one, so a machine that builds linux/amd64 itself is preferred.

A step that fails on one of your machines carries a failure code in its reason, and the failure report says what it means:

Code Meaning Retried
registry_auth_failed The registry refused the machine’s credential during an image build. A fresh credential is issued on the next attempt. Yes
artifact_verification_failed An input artifact did not match its signed digest, so the machine refused to use it. No
unsupported_feature The machine lacks a feature the step needs (an older agent, no Rosetta lane, no egress filter). Another machine is tried when there is one. Elsewhere
platform_mismatch The step needs a platform the machine does not run. No
invalid_spec The machine refused the step’s specification. No

A step that only prints one of these words never changes how its failure is classified: a command that exits non-zero is your code.