Skip to content

Native Groves

A native Grove is a machine you own, a Mac with Apple silicon or a Linux machine with Docker, that runs Canopy’s agent and takes compute work from your team: native jobs, accelerator jobs and, on a Mac, native workloads. It needs no VPN. Everything it does is an outbound HTTPS connection to Host. A Mac runs a step as a host process with its Apple GPU; a Linux machine runs every step in a Docker container, with its NVIDIA GPU (Native Linux nodes, Native Mac nodes).

The routes sit behind the host.native_groves platform feature, which is on for every team. If it is switched off, the team routes below answer 403 with { "error": "feature_disabled", "feature": "host.native_groves" }. canopy forest join is the client these routes are built for; the same calls work with curl and an API key.

Every Grove enrolled through these routes has the same fixed shape. You cannot change it at enrollment:

Property Value
Substrate native (platform darwin/arm64, linux/amd64 or linux/arm64)
Roles runtime and build
Placement team: work of your team that names no machine may run on it, besides work that names it. You can limit it to work that names it (explicit, see below)
Availability intermittent: a laptop that sleeps is expected to disappear
Owner your team
Region personal, a pseudo-region with no jurisdiction (created on first use)
Transfers host_brokered (see below)

A native Grove runs trusted work of its own team. It is not a sandbox for untrusted code.

Team → Machines lists the team’s machines: state, the agent version (with the newest published release when yours is behind), the operating system, Docker and NVIDIA driver versions a Linux machine reported, its GPUs, the result of the last check, whether steps that name no machine may run there, and its pools.

  • Add a machine shows the one command to run on the machine (canopy forest join --team <team> [--name <name>] [--no-auto-placement]) and what the machine needs: a Mac on Apple silicon, a Linux machine with Docker, or one with an NVIDIA GPU as well. I ran it, wait for the machine watches the list until the machine appears. The command signs the machine in through the CLI; the app never handles an enrollment token.
  • Diagnose opens what Canopy knows about the machine: its state, how long ago it was heard from and last asked for work, whether its identity is pinned, and the last ten checks with the detail of each one that failed. Nothing secret is shown.
  • Revoke asks first and lists the steps that are running or waiting on the machine, which fail when it goes. It then removes the machine from the team. Adding it back means running the join command again.

The buttons that change something need the grove:manage action; looking needs grove:read.

Needs the grove:manage action (the admin tier by default).

POST /teams/{team_id}/groves/native/enroll-tokens
{ "name": "gio-macbook", "ingress": false }
{
"enrollment": {
"token": "5b0e...",
"expires_at": "2026-10-08T12:15:00.000Z",
"grove_name": "gio-macbook",
"region_id": "00000000-0000-4000-8000-0000000000e2",
"api_url": "https://api.canopy.pm",
"host_cert_fingerprint": "9f2c...",
"host_public_key": "-----BEGIN PUBLIC KEY-----\n...",
"env_class": "production",
"release_manifest_url": "https://api.canopy.pm/forest/agent/releases/latest?platform=darwin-arm64",
"ingress": false
}
}
  • The token is shown once, is valid for 15 minutes and works once. Minting again for the same name replaces your own unused token for it.
  • name is 1 to 63 characters of a-z, 0-9 and -. Names are unique across Canopy: a name that exists, in any team, answers 409 with error_code: grove_name_taken. A team may hold 10 native Groves (unused tokens count); more answers 409 with native_grove_limit. A revoked Grove frees its place, and its name can be reused by the same team.
  • host_cert_fingerprint and host_public_key come from the authenticated answer. The agent pins both when it enrolls and refuses an unpinned production enrollment. The same public key verifies the release manifest.
  • ingress is echoed back for the installer. Nothing is provisioned for it yet.

The agent then calls POST /forest/groves/enroll with the token, like any other Grove.

All of these are scoped to your team. A Grove of another team, or a Kubernetes Grove, answers 404.

Route Needs Answer
GET /teams/{team_id}/groves/native grove:read (every member) { "groves": [...] }: id, name, state, version, last heartbeat, capabilities (roles, platforms, accelerators, the features the agent proved and the change_kinds it can run, machine state including the operating system and the Docker and NVIDIA driver versions a Linux machine reported), last_verification and latest_agent_version (the newest published agent for the machine’s platform, null when none is published). Revoked Groves are not listed.
GET /teams/{team_id}/groves/native/{grove_id}/diagnose grove:read A redacted bundle: state, heartbeat and poll age, capabilities, configuration presence booleans and the last ten verification records and active_jobs (at most 20, with active_jobs_truncated): the jobs running or waiting on the machine, which revoking it fails. Never a token, a key or an environment value.
DELETE /teams/{team_id}/groves/native/{grove_id} grove:manage 204. Revokes the Grove’s unused enrollment tokens and the Grove itself; it stops authenticating at once. Calling it again is also 204. The agent on the machine clears its job cache the first time Host answers that the Grove was destroyed (agent 0.1.9 and later).
PATCH /teams/{team_id}/groves/native/{grove_id} grove:manage { "auto_placement": true | false, "pools": ["name", ...] }, either or both. 200 with the Grove, whose auto_placement and pools show the settings. See Automatic placement and Pools. 400 when neither is sent (invalid_update), for a non-boolean (invalid_auto_placement) or for a bad pool (invalid_pools).
POST /teams/{team_id}/groves/native/{grove_id}/verifications grove:manage 201 with the stored record.

A pipeline step that asks for a native machine can name one (native.grove). A step that names none, or a GPU container step, goes to whichever machine can take it. A joined machine takes such work by default (auto_placement: true, placement policy team); opt it out with canopy forest join --no-auto-placement or canopy forest groves update <name> --auto-placement off and only steps that name it run there. Both are recorded as grove.policy_changed.

Canopy considers the machines of your team that run the step’s platform, offer the GPU it asks for and are awake, and, when the job allows Canopy’s capacity, the machines offered to every team. It prefers a machine of your team over a Canopy machine, then the one with the smallest GPU that is big enough (so a big card stays free for a big job), then the one holding fewer GPU jobs, then the one it heard from most recently. A machine that is asleep, draining or busy makes the step wait, shown with the reason (machine_asleep, gpu_busy, and the others listed under compute jobs), rather than fail. A GPU step no machine fits yet waits as no_fitting_gpu, because a fitting machine may join. A step that could never run anywhere that is not a GPU step fails at once and names why.

Neither way ever runs a pull request from a fork on a native machine.

In canopy.pipeline.ts, leave grove out of the step’s native field to let Canopy choose (@canopy/ci 0.21 and later):

step('train', {
native: { platform: 'darwin/arm64' },
run: 'python train.py',
resources: { accelerator: { type: 'gpu', vendor: 'apple', memoryMb: 4096 } },
})

A step that would rather fail than queue says so with unavailable: 'fail' (the default is 'wait'):

step('train', { native: { platform: 'darwin/arm64', unavailable: 'fail' }, run: 'python train.py' })

With fail, a step finding every fitting machine busy or away fails at once and names who was busy or away, instead of waiting up to the scheduling window. It applies to a step that names its Grove as well.

To see where each native or GPU step of a graph could run right now, without running anything, use canopy pipelines plan <graph_id|file>. It prints one line per step: who is ready, who it would wait for, and who is refused. Add --verbose to get one line per machine with the reason, for example old-mac - not_opted_in or studio-mac - incompatible:accelerator_unavailable (has no GPU).

A machine can build the image of a Dockerfile or buildpack deploy for your team instead of Canopy’s build machines, and a service can build for linux/arm64 as well as linux/amd64. This is a preview behind the host.native_offload platform feature and your team’s execution policy, and nothing is sent to a machine until the owner of that machine says so when joining it:

canopy forest join --accept image-builds

Nothing is accepted by default. A build that uses variables also needs --accept secrets.

The build runs in a rootless BuildKit container on a Linux machine, and in a Linux guest made for that one build on a Mac. The network is open: the Dockerfile can reach anything the machine can reach, and there is no egress proxy on a machine, unlike a Canopy build machine, which limits a build to the hosts it declared. The machine pushes only the image of the build it runs, its own build cache and your team’s dependency images; it cannot read another team’s images, cannot delete anything and never receives a pull request from a fork. An image built on a machine is recorded with the machine that built it, and is checked to be built for the platform it was asked for before it is used.

canopy environments update <environment_id> --build-platforms linux/amd64,linux/arm64 (or none) makes the build one image for each platform and publishes them as one image, and step.buildImage({ platforms: [...] }) does the same in a pipeline. Canopy has no arm64 build machine of its own, so a build for linux/arm64 needs a machine of your team that builds it, and the plan fails before anything builds when there is none. A deploy runs the image that fits the machine it lands on. canopy builds show lists the image of each platform and where it was built.

Host’s /metrics carries the state of the team machines it knows about, all as gauges refreshed at most once a minute:

Metric Meaning
canopy_native_groves{state} Native Groves that are ready, away (no report for five minutes), draining or in maintenance.
canopy_native_groves_opted_in Native Groves that accept automatic placement.
canopy_native_jobs_24h{status} Jobs created on native Groves in the last 24 hours, by status.
canopy_native_job_refusals_24h{code} Jobs an agent refused to start, by refusal code (busy, platform_mismatch, …).
canopy_native_jobs_unacknowledged Jobs the agent has not acknowledged a minute after they were committed; Host is delivering them again. A value that stays above zero means an agent is not answering.
canopy_native_placement_wait_seconds{quantile} How long steps waited, in the last 24 hours, between starting and their job being committed to a machine.

A pool is a label you put on some of your machines so a step can choose among just those. Label a machine with canopy forest groves update my-mac --pools cortex-gpu (or pools on the PATCH route, or the Pools field on the Machines page), and ask for the label in the pipeline:

step('train', { native: { platform: 'darwin/arm64', pool: 'cortex-gpu' }, run: 'python train.py', resources: { accelerator: { type: 'gpu', vendor: 'apple' } } })

A step with a pool is placed like any automatic step, among the machines that opted in to automatic placement, and then only those carrying the label. A second Mac you label and opt in is used without editing the pipeline. A pool never replaces what the step needs: it must still declare the GPU it wants. pool and grove cannot be set on the same step (native_target_conflict). A pool name is 1 to 32 lowercase letters, digits or hyphens, and a machine has at most 8 pools. canopy pipelines plan shows a machine outside the pool as refused with not_in_pool. Changes are recorded in grove.policy_changed. Needs @canopy/ci 0.24 or later.

Canopy staff can offer a Linux machine its owner opted in to automatic placement to every team (arborist groves policy <id> --offered on). Only a machine that runs Linux containers can be offered; a Mac runs host processes and is never offered. Offering a machine withdraws itself when its owner switches automatic placement off.

A team other than the owner reaches an offered machine only with a job that allows Canopy’s capacity (providers includes canopy), only with an image from Canopy’s platform registry namespace, and only while its plan’s GPU allowance has room: one GPU per run and ten GPU-hours a month, hard, on every plan, which staff raise per team with arborist accelerator-entitlements set. A team without allowance waits as not_entitled, a team that used its month as over_budget. The seconds are charged to the team that submitted the job, never to the owner; the owner’s own jobs on their own machine are free. If the allowance runs out between the choice and the start, Canopy gives the machine back and the step waits again.

An app service can be bound to the machines it may ask Host for. A binding names one machine and what the service may use it for: accelerator or runner (jobs) or inference, with a priority from 0 (tried first) to 9. At most 8 bindings per service.

Terminal window
curl -X PUT https://api.canopy.pm/environments/$ENV/app-services/$SERVICE/grove-bindings \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"grove_bindings":[{"grove_id":"<machine id>","role":"accelerator","priority":0}]}'

or sdk.appServices.setGroveBindings(env, service, bindings). A machine of your team is always bindable, whatever its placement policy, so a machine kept to named work (--no-auto-placement) can still serve the service. A machine of another team is bindable only when Canopy offers it and the role is not inference. Binding a machine that does not exist or was destroyed is a grove_not_available error and stores nothing.

A service with at least one binding gets an identity of its own, which can only submit compute jobs and read them. A job it submits with that key must name placement.role, and is placed on the machines bound to it for that role first, then on the team’s machines that take unnamed work, then on Canopy’s capacity (when providers allows it). It cannot pin a machine of its team that is neither bound to it nor open to unnamed work. Removing the last binding switches the identity off at once; its next request is refused. Bindings are not part of an environment backup, because a machine belongs to the team that enrolled it.

An inference binding can name the workload on the machine that serves the model (endpoint, lowercase letters, digits and dashes; canopy app-services grove-bindings <service> --set <machine id>:inference:0:<workload>; the workload is the one canopy’s machine view lists, named w-<environment id>, not a process name such as web). The binding then lists an endpoint_id, and the service calls GET /grove-endpoints/<endpoint_id> with its own key to learn whether the machine can serve (running, degraded, starting, failed, missing, sleeping, offline or unreachable, with the seconds worth waiting) or sdk.groveEndpoints.status(endpointId). An OpenAI-compatible client reaches the model at /grove-endpoints/<endpoint_id>/v1/... (/v1/models, /v1/chat/completions, /v1/completions, /v1/embeddings, /health; GET and POST only, bodies up to 8 MiB, eight requests at once per binding). While the machine is asleep, offline or starting, the answer is 503 with retry_after_seconds; a machine that does not answer in time is 504. The request reaches the machine through the tunnel its agent holds to Canopy, so a machine whose agent predates the tunnel answers 503 endpoint_unreachable until it is updated with canopy forest update.

An environment can start from a registered bundle, with no repository, image or template:

Terminal window
canopy environments create --project <project_id> --native-bundle <artifact_id> --grove <grove_id> [--config-file ./canopy.yml]

or sdk.environments.create({ projectId, nativeBundleArtifactId, groveId, config }), or POST /environments with native_bundle_artifact_id and grove_id (a setup request takes the same two fields per environment). The machine is required, belongs to your team and must run the bundle’s platform; the environment takes its region from it, and its first deploy starts at once. config is the content of a .canopy.yml (the same document an environment reads from a repository); without it a bundle with one entrypoint runs that entrypoint as its web process. The environment has one service; it cannot be promoted, previewed or exported to a template, and its Blueprint source is read-only (native_bundle_source_read_only). Deploy another bundle with canopy deploys create --env <id> --native-bundle <artifact_id>; the environment then points at that bundle, and a rollback moves the pointer back.

Creation is refused with 409 native_bundle_source_conflict when a repository, image or template is also given, 422 grove_id_requires_native_bundle or config_requires_native_bundle when the field has no bundle, 422 native_bundle_processes_required when no process can be derived, 422 native_compile_failed (with violations) or 422 config_invalid when the configuration does not compile, and by the bundle checks listed below.

A native workload runs a bundle: an archive with a manifest.json (schema, platform, entrypoints) and the files it names. Register the bundle as an artifact of your team and deploy it on an environment whose machine is native:

Terminal window
canopy artifacts blob put ./bundle.tar.zst --team <team_id> --type native-bundle --name my-model --platform darwin/arm64
canopy deploys create --env <environment_id> --native-bundle <artifact_id>

or sdk.builds.create(environmentId, { nativeBundleArtifactId }), or POST /environments/{environment_id}/builds with native_bundle_artifact_id. The deploy appears in the environment’s deploy history like any other, with job_kind native_bundle and the bundle it ran. It is refused with 404 native_bundle_not_found for a bundle of another team or an artifact that is not a bundle, 409 native_bundle_not_native_environment for an environment whose machine is not native, 409 native_bundle_grove_unavailable for a machine that is draining, in maintenance, unavailable or whose agent does not advertise native workloads, and 422 native_bundle_platform_mismatch or 422 native_bundle_manifest_invalid when the bundle cannot run there. The same checks run again when the worker picks the deploy up. A rollback to an earlier deploy redeploys that deploy’s bundle and configuration as a new deploy (409 native_bundle_unavailable if the bundle no longer exists).

The archive must be a zstd-compressed tar with manifest.json at its root; the upload reads it, takes the platform from it and refuses (400 native_bundle_manifest_invalid, native_bundle_manifest_missing, native_bundle_archive_invalid or native_bundle_platform_conflict) an archive without a valid manifest or whose --platform disagrees with it. A process of the workload is a manifest entrypoint of the same name (web by default), and its arguments come from the manifest: args in .canopy.yml is refused for a native process. The workload listens on the PORT the agent passes and is the one an inference binding names.

An Apple GPU shares the machine’s memory, so memoryMb on the accelerator is a capacity check against the memory the machine reports, not a reservation: Canopy refuses a step that asks for more than the machine has, and the agent refuses to start one that would not fit beside what is running, but two steps cannot be told apart by the hardware. Only one GPU step runs on a machine at a time; a second waits for the first. The machine’s other cores keep running ordinary steps meanwhile.

A laptop that sleeps stops reporting. Canopy treats a machine as away after five minutes without a report, and a step that could only run there waits rather than being queued on it (nothing is committed to a machine that might not wake). A step that was already running when the machine went to sleep is retried once the agent restarts. Keep the machine on power with sleep prevented for long runs.

A verification record is the result of the checks join runs after installing:

{
"agent_version": "0.1.0",
"at": "2026-10-08T12:03:11.000Z",
"checks": [
{ "name": "enrolled", "ok": true },
{ "name": "metal", "ok": true, "detail": "Apple M3 Max, 49152 MiB unified" },
{ "name": "mlx", "ok": false, "detail": "mlx bundle not installed yet" }
]
}

checks holds 1 to 32 entries with unique lowercase names (enrolled, heartbeat, tunnel, native_job, metal, mlx and mesh are the ones the installer reports), detail is at most 500 characters and at may not be more than five minutes ahead of Host. The record is ok when every check is. Host keeps the newest 50 per Grove. The Grove can post its own record, signed like its other requests, to POST /forest/groves/{id}/verifications.

A machine updates its agent only when its owner says so. The setting is agent_updates on the Grove: off (the default), stable or latest, set with PATCH /teams/{team_id}/groves/native/{grove_id} and needs grove:manage. A release starts on the latest channel; Canopy promotes it to stable once it has held up, so a machine on stable only sees promoted releases and a machine on latest sees every release. A withdrawn release is never offered.

A machine on a channel is offered a release only when it is newer than the one it runs and built for its own platform; a machine is never moved to an older version this way. The offer is the agentUpdate field (version, channel) of its poll reply. The agent then asks POST /forest/groves/{id}/agent-update, signed like its other requests, which returns the same Host-signed manifest that canopy forest update verifies, for exactly that release. The agent checks the signature against the Host key it pinned at enrollment, the sha256 and size of every file, and waits until no job is running. canopy forest update on the machine keeps working whatever the setting says.

An agent that accepts image builds proves each lane by building a one-line image in the real builder before it offers the lane, and the builder and base images live in platform-registry/, which refuses an anonymous pull. The agent asks Host for a login with POST /forest/groves/{id}/registry-credential, signed like its other requests. The answer holds the registry_host, a username and password, and expires_in_seconds (900). The login can pull platform-registry/ and nothing else: it never pushes, and it never reaches a team’s images, caches or dependency images. It is used for that machine’s preflight only, kept out of the preflight’s files once it ends, and a machine whose request fails runs the preflight without one, so its lane is simply not offered. Real builds keep using the login Host puts in each build’s secret, which the builder image pull now uses too.

The login is accepted by the public registry host (registry.canopy.pm) as ordinary Basic credentials: the registry front checks its signature and expiry and allows exactly what that kind of login may do (a preflight pulls platform-registry/; a native build pushes and pulls its own forest-builds/<build> repository). Customer API keys keep working there unchanged.

The Linux agent and the Mac agent are two programs built from one codebase. A Linux agent runs steps and builds images in containers of its own Docker (BuildKit rootless, as an unprivileged user) and updates itself as a user service. A Mac agent runs a step or an image build in a throwaway Linux guest of its own and updates itself through its root helper. Neither binary contains the other’s code, and the release of a platform never depends on what the other platform’s build left behind.

A Mac builds an image in a Linux guest. The guest has the capabilities BuildKit needs to mount its snapshots (it is a virtual machine of its own, so they reach nothing of the Mac), keeps BuildKit’s state on its own disk, and has the open network through the Virtualization framework’s NAT. It resolves names through the servers the Mac itself uses. From inside the guest the network reaches whatever the Mac’s network reaches, home network included, like the open network of a Linux machine. The agent offers portable_vm_network_v1 for a platform only after a guest of that platform fetched a page in its preflight.

The rootless BuildKit keeps its state in its own user’s directory (the agent passes --root), the builder’s own pull of a base image uses a read-only registry login the agent mounts for the preflight, and a build that Host places on the machine needs the builder image Host pins for it: the Host environment variables FOREST_NATIVE_BUILDER_DOCKERFILE_ROOTLESS, FOREST_NATIVE_BUILDER_DOCKERFILE_VM, FOREST_NATIVE_BUILDER_BUILDPACK_ROOTLESS and FOREST_NATIVE_BUILDER_BUILDPACK_VM (operators set them to the digests the agent release lists under builders). Without them no machine is eligible and the placement only says a feature is missing.

An image build for an environment on build type auto also needs the buildpack lane, so a machine that proved only the Dockerfile lane takes Dockerfile environments (build_type: dockerfile). The image platform is the environment’s build_platforms; a Mac builds linux/arm64, a Linux PC its own architecture.

GET /forest/agent/releases/latest?platform=darwin-arm64
Authorization: Bearer <API key, or an unused native enrollment token>
{
"manifest": {
"platform": "darwin-arm64",
"version": "0.1.0",
"min_macos": "14.0",
"binary": { "url": "https://api.canopy.pm/forest/groves/release/transfers/...", "sha256": "<hex>", "size": 123, "signed": "adhoc" },
"helpers": [ { "name": "wireguard-go", "url": "...", "sha256": "<hex>", "size": 1 } ],
"bundles": [ { "name": "mlx", "version": "x", "url": "...", "sha256": "<hex>", "size": 1, "requires": "accelerator:apple" } ],
"released_at": "2026-10-08T10:00:00.000Z"
},
"signature": "<base64>"
}

The trust root is Canopy’s own key, not Apple’s. Binaries are ad-hoc code signed (signed: "adhoc"); a Developer ID release would say developer-id and nothing else about the flow changes. A linux binary carries no signature of its own (signed: "none", and min_macos is 0.0, which a linux client ignores); its sha256 in the signed manifest is all its trust. A client must:

  1. Take manifest exactly as parsed, write it as canonical JSON (object keys sorted at every level, no whitespace outside strings), and verify signature (Ed25519, base64) over those bytes with the host_public_key it was given at enrollment.
  2. Download each file from its url and check the sha256 and size before running anything.

The URLs are short-lived (15 minutes), so the manifest is signed again on every request; the digests are what stays trusted. The answer is 404 with error_code: no_release when nothing is published for the platform, and 400 for a platform that does not look like darwin-arm64. A client that downloads the files itself, rather than a browser, never sets the macOS quarantine flag, so Gatekeeper is not involved.

A Grove normally uploads and downloads workspaces and bundles through presigned URLs against the object store, which sits on a private network. A native Grove on a home network cannot reach it, so Host carries the bytes: a Grove whose workspace_s3_mode is host_brokered gets URLs of the form

https://api.canopy.pm/forest/groves/{grove_id}/transfers/{token}

instead. The token is signed by Host, names one object and one verb (GET or PUT) and expires. No other credential is involved, so curl works. The agent release files use release in place of the Grove id.

Behaviour Detail
Streaming The body is streamed in both directions and never held whole in memory.
Size A PUT needs a Content-Length (411 otherwise) of at most FOREST_TRANSFER_MAX_BYTES, 2 GiB by default (413). A GET of a larger object is 413 too.
One upload per token A second PUT with a token that already stored its object is 409 transfer_token_used. An upload that fails or is cut off gives the token back, so it can be retried until it expires.
Concurrency A Grove may have 4 transfers in flight (FOREST_TRANSFER_MAX_CONCURRENT_PER_GROVE); more answer 429 with Retry-After: 5.
Lifetime Tokens for the release files and for bundles last 15 minutes. A token for a job’s workspace upload or seed lasts as long as the presigned URL it replaces would have (the job’s timeout plus a margin, at most 24 hours), because the upload happens when the job ends.
Errors 401 for a malformed, forged or expired token, 403 for a token used on another Grove’s path or with the other verb, or for a Grove that is not host brokered, 404 for an unknown Grove or object.

Groves enrolled any other way keep direct transfers and their presigned URLs, byte for byte as before.

Action Default tier Covers
grove:read viewer listing and diagnosing the team’s native Groves
grove:manage admin minting tokens, revoking, posting verification records

Both can be granted to a custom role. See Your permissions.