Skip to content

Deploys & the Staged Pipeline

Canopy runs a pipeline for each deploy. Its exact operations depend on the environment and its pipeline configuration. A Git-backed deploy checks out the selected commit, builds an image, runs any required approval and pre-deploy release tasks, then rolls out the services and checks their health. The build page reports the failing stage and reason.

The configured build_command is part of the image build configuration. Configured build_steps run later as ordered release tasks, each in its own task using the built image. They gate the deploy, but changes made in one task’s container are not shared with the next task. The previously active release remains in place if a release task fails.

Every build has a number within its environment: 1, 2, 3 in creation order, never reused, with gaps possible if a build row is removed. It is shown as #142 in the app and in canopy deploys list / canopy builds show, and it is the number field of every build in the API (null on a build that has not been numbered yet). A preview environment numbers its own builds.

Wherever a route takes a build under an environment (/environments/{id}/builds/{build_id}... and /environments/{id}/deployments/{build_id}), build_id may be the UUID or the number, 142 or #142 (URL-encoded as %23142). A string of digits (no leading zero, at most nine digits) is always a number and a UUID never is, so the two cannot be mixed up. A number the environment has no build for is a 404 with error_code: "build_number_not_found"; a number is never looked up across environments. The CLI accepts 142 or '#142' (quote it: a shell reads an unquoted # as a comment) for the commands that already need --env.

A build becomes successful when its build pipeline and rollout complete: the new web and worker services have converged and the runtime accepts the release. A build held for human review is awaiting_approval. A terminal rollout failure is reported on the build; a separate public HTTP check does not hold the build open.

After a successful deploy, Canopy queues a bounded public verification for web services whose public check is enabled and which have a domain. It probes the configured health path up to six times over about 90 seconds and records healthy or unhealthy separately from build status. No configured domain or a disabled public check records unknown; an environment with no web service records not_applicable. This verification never delays or fails a deploy. See Environments & Services for service health settings.

The deployment read (GET /environments/{id}/deployments/{build_id}) reports the verification as status, attempts, checkedAt, probeTarget (the address probed) and detail (what the probe saw, for example GET /health: no response within 30 s); the last two are null when the probe recorded none.

Status Meaning
successful Built, deployed and healthy.
failed A step failed or the pipeline could not complete.
deploy_failed CI passed but the deploy did not become healthy.
timed_out The pipeline or the deploy did not finish in its time budget.
errored An internal Canopy error.
cancelled / rejected Stopped by a person, or replaced by a newer build.

canopy builds cancel <build_id> --env <environment_id> cancels one pending or running build: its unfinished steps and Grove jobs are stopped and no other build is touched. Asking again for a build that is already cancelled answers 200 with already_cancelled: true and changes nothing; a build that already succeeded or failed answers 409. The command reads the environment’s slug itself for the confirmation described below.

A failed build carries a failure object (canopy builds show prints it, and so do the API and MCP) saying where it stopped and whose problem it is:

  • stage: source, pipeline, artifact, capacity, admission, release_task, rollout, routing, tls, health_check, deploy_timeout or internal.
  • kind: user_code, user_config, platform, infrastructure, capacity or timeout.
  • detail: the specific reason, for example app.example.test: hostname already claimed or HTTP 502.

Next to failure, a failed build detail carries an error: the failure report that ended the build. Where failure says which stage stopped, error says what happened to which node or generation, whose problem it is and whether trying again can help:

  • reference: an fr_… value to quote to support. It identifies one failed attempt.
  • category: source_fetch, configuration, build_step, test_failure, image_build, release_task, rollout, readiness, capacity, infrastructure, platform_defect, external_service, timeout or unknown.
  • owner: customer, canopy, external or unknown. retryable: yes, no or unknown.
  • operation: the node_id, node_attempt, generation and release_id the failure belongs to.
  • message and detail (the exit code and an excerpt of the failing output) are present when the failure is yours. When Canopy owns the failure (owner: canopy) they are null and public_message is the explanation; quote the reference and Canopy staff can read the technical details.

error is null for a build that did not fail, for a failure Canopy later withdrew because the deploy converged, and for builds that ran before failure reports existed. Every node in execution.nodes also has an error with the latest report of that node, so a node that failed and then succeeded on a retry still names the attempt that failed (outcome: recovered).

canopy builds failures (the API routes GET /environments/{environment_id}/builds/{build_id}/failures and .../failures/{reference}) lists every report of a build, oldest first, including attempts that recovered. A build keeps at most 32; dropped counts any refused past that. A reference that belongs to another build is a 404, the same as one that does not exist. Output excerpts are masked with the build’s secrets and with common credential patterns, and are kept for 90 days; the report itself stays with the build.

A build detail returns at most about 512 KiB of build_logs, the newest rows. When the log is larger, build_logs_omitted says how many of the oldest rows were left out; it is 0 for a build whose log fits. The log stream and the retained log are not affected.

A certificate or route that is still pending does not by itself fail a rollout. A terminal runtime refusal or a timeout does. deploy_failed and timed_out builds can be retried like a failed one. The later public verification result is separate from these build statuses.

An environment can depend on others when its build needs theirs finished first: an app that consumes a library built from the same repository, for example. canopy environments depends-on <environment_id> --on <environment_id> makes the builds of the first wait for the builds of the second for the same commit; --remove <environment_id> stops that, and canopy environments dependencies <environment_id> (the API route GET /environments/{environment_id}/dependencies) shows both directions. The App has the same under the environment’s settings, “Build dependencies”.

  • A build waits while a build of an environment it depends on, for the same commit, is pending or running. canopy builds show and the build page say what it is waiting for.
  • If one of those builds failed, errored, timed out or could not fetch its source, the waiting build ends as failed without running and says which build failed, so it is never built against a broken dependency. A cancelled upstream build does not hold the build back or fail it.
  • Nothing to wait for means no waiting: if the dependency has no build of that commit, the build starts at once.
  • An upstream build created more than two hours ago no longer holds a build back, so a stuck build cannot stall the queue.
  • Dependencies are between environments of the same Organization, at most 20 per environment, and a cycle is refused. Builds of environments built from different repositories have different commits, so they never wait for each other.

canopy builds show prints an efficiency section, and the build detail carries it as phase_timings and execution.nodes[].measurements:

  • phase_timings lists how long Canopy spent generating the pipeline graph (phase_generate), planning it (phase_plan, with the time of each planning step such as execution_keys and policies) and promoting the built image into the platform registry (phase_promote, with the blobs and bytes it copied and the blobs the registry already had). A build that did not do a phase has no entry for it.
  • Each node’s measurements hold the scheduling, start-up and run time of its Pod and, for steps and image builds, the command time, peak memory and CPU the container used. A figure the machine did not report is absent, not zero.

Image-build figures appear for builds whose build worker image carries the reporting; older worker images report none.

canopy builds steps <build_id> --env <environment_id> (the API route GET /environments/{environment_id}/builds/{build_id}/telemetry) shows where each job of a build spent its time, as the build worker measured it:

  • Phases: the steps BuildKit ran, the layers it pulled and unpacked and what it exported, each with how long it took and whether it was cached. --all lists every phase; by default the slowest twelve per job.
  • Counters: the worker samples the kernel’s cumulative counters every few seconds while it runs and the summary is the difference between the first and last sample: CPU used and throttled, time spent waiting on I/O, CPU or memory (pressure stall), disk and network bytes, peak memory. --samples also lists how the counters changed between samples.

A figure the kernel does not give is shown as “not reported by this kernel” and is null in the API; it is not zero. A job that reported nothing, because its worker or controller predates measurement, says so. --node <node_id> narrows to one job and --json prints the jobs, phases and samples.

canopy builds timeline <build_id> --env <environment_id> draws each job on one time axis: how long it waited to start (.) and how long it ran (#), so the longest wait or the job on the critical path is visible at a glance. A job from an older API that sends no queue time is drawn from its start. --json prints the times.

canopy builds logs <build_id> --env <environment_id> prints the stored log of a build; --node <node_id> prints only the log of one node of its pipeline and names the nodes there are when the one you gave does not exist.

canopy builds activity <build_id> --env <environment_id> (the API route GET /environments/{environment_id}/builds/{build_id}/forest-activity) lists, oldest first and in your local time, what Host and the Grove did for the build: jobs handed to the Grove, waits for capacity with the scheduler’s own reason (for example Insufficient cpu), and pod events. A job that sits in running with no matching line from the Grove is the sign that it never reached the Grove. Host fails such a job after 15 minutes without any report from the Grove (lost: grove ... never reported on this job), instead of leaving the build waiting. --json prints the lines.

Builds Canopy runs itself start one at a time: while one is in flight, the next stays pending until it finishes (BUILD_MAX_LIVE_BUILDS, default 1), because a build Grove sized for one large build cannot schedule two.

Dependency images: reused, built or a fallback

Section titled “Dependency images: reused, built or a fallback”

A step that declares dependencies gets an image with them installed once per lockfile (see the pipeline reference). The build detail carries what became of each such step as dependency_resolutions, and canopy builds show prints one line each:

  • reused: a stored image, or one another step was building at the same time, was used.
  • built: a new image was built for this step.
  • fallback: no image was provided and the step installed its own dependencies. This is not the optimisation working. The reason is one of UNSUPPORTED_STEP, MACHINE_BUILDS_NOT_ACCEPTED (the step runs on one of your own machines and its owner has not allowed image builds there), DRIVER_UNSUPPORTED, NO_BUILD, UNTRUSTED_RUN, NO_SOURCE, TREE_UNAVAILABLE, NO_PACKAGE_MANAGER, NO_LOCKFILE, IMAGE_CHECK_UNKNOWN, NO_ENVIRONMENT, IMAGE_BUILD_FAILED (with what the build said in detail) or ERROR.

The same line is written at the top of the step’s log, and /metrics counts them as canopy_dependency_resolutions_total{outcome,reason} and canopy_dependency_resolution_seconds_total.

Every build also records a release: the image each service ran and the configuration it was built with. A release is building while its build runs, built once it succeeds, and active once it is live. The release that was live before it becomes superseded. A build that fails or is cancelled leaves a failed or cancelled release and never changes which one is live.

GET /environments/{id}/releases
GET /environments/{id}/releases/{release_id}
POST /environments/{id}/releases/{release_id}/rollback

Each release lists its services (image digest reference and a snapshot of the service settings) and the service_builds that produced it; the build detail response carries the same release and service_builds. Rolling back to a release creates a new rollback build, exactly like rolling back to the release’s build, and records which release it restores. Only a release that was live can be restored (release_not_rollbackable), not the active one (release_already_active), and only while the environment still has the services it had then (release_service_set_mismatch).

Rolling back (to a build, to a release, or to the last healthy build) and cancelling a build need the environment’s slug as confirm, in the JSON body or the query string, the way deleting a project needs its slug. Without it the answer is 409 with error_code: "confirmation_required", which names the slug to send:

{
"code": "conflict",
"error_code": "confirmation_required",
"title": "This will roll back environment \"production\". Pass confirm=production to go ahead.",
"confirm_field": "confirm",
"environment": { "id": "5f0e...", "slug": "production" }
}

The app, the CLI, the SDK (confirm argument) and the MCP tools send it for you.

Rolling back safely when the environment may have moved

Section titled “Rolling back safely when the environment may have moved”

The app’s single Roll back button opens a picker: the last verified-healthy build, or any release that was live and has been replaced. It sends the live release it saw as expected_current_release_id, so a push that landed in between reloads the list instead of rolling back the wrong thing.

A rollback dialog shows the release that is live now and the one it will restore. Between the moment you looked and the moment you confirm, someone else (or a push) can deploy something new, and the rollback would then undo that instead. Every rollback endpoint therefore accepts an optional precondition:

POST /environments/{id}/releases/{release_id}/rollback
POST /environments/{id}/builds/{build_id}/rollback
POST /environments/{id}/builds/rollback (the newest build verified healthy)

Send expected_current_release_id (the environment’s active release as you saw it, or null for none) and/or expected_current_build_id (its current build), in the JSON body or the query string. If either no longer matches, nothing is rolled back and the answer is 409 with error_code: "release_changed":

{
"code": "conflict",
"error_code": "release_changed",
"title": "The environment changed since you looked at it: its current release is no longer the one you expected.",
"current_release_id": "5f0e...",
"current_build_id": "9a21...",
"retryable": false
}

Show the new state, let the user decide again, and retry with the new ids. The check is made again when the rollback build is created, under a lock on the environment, so two rollbacks carrying the same expectation cannot both be accepted. Omit both fields to keep the previous behaviour. A value that is not a UUID is a 400.

Several services: one release, all or nothing

Section titled “Several services: one release, all or nothing”

When an environment has several services, each is built from its own folder and the release goes live only when every web and worker service has converged on it. If one fails, the services already updated are put back on the previous release and the release is reported reverted (or failed when there was nothing to go back to), with the service that failed and where. While a release is rolling out, services can briefly serve different releases: Kubernetes cannot switch several deployments at one instant. Jobs never block a release.

Rolling back to a release of a multi-service environment redeploys the images that release was made of, all at once, without building anything. If the build worker restarts while a release is rolling out, the release is settled within a couple of minutes from what the services report.

canopy build-steps add "npm ci"
canopy build-steps add "npm test"
canopy build-steps list --env <environment_id>

build_steps run in order as separate pre-deploy tasks in the built image. Use them for checks or commands that need the image and its runtime configuration; they do not share filesystem changes with later steps. For example, if the image includes a pre-deploy check script:

canopy build-steps add "node scripts/predeploy-check.mjs" --env <environment_id>
canopy build-steps list --env <environment_id>

Use the image builder or Dockerfile to install dependencies and produce build outputs. A single --build-command on environments update configures the build command; it is not a replacement for the ordered release tasks.

Set --build-path (on environments create/environments update) to build from a subdirectory instead of the repo root - this is also how multi-service environments let one repo’s frontend and backend deploy independently.

A push builds an environment only when it changed the environment’s build_path. Canopy reads that from the file lists in the push payload, except when the payload cannot be trusted: it was cut short (GitHub and GitLab list at most 20 commits), or it holds a merge commit, whose file list is taken against its first parent and not against the branch head the push moved from. In those cases Canopy compares the branch’s previous head with its new head itself, so pulling before you push does not rebuild the environments the other side of the merge had already built. When that comparison cannot be made (a new branch, a head the provider cannot serve) the environment builds, because skipping a real change is worse than one extra build.

A pipeline can name more paths than the build path with watch in its canopy.pipeline.ts (a shared SDK, a script, committed baselines the build reads; see Canopy Pipelines). The environment then also builds when a push touches one of them, and a skipped push (skipped_path in the webhook deliveries) lists them as watch_paths next to the build_path. Host stores the list on the environment as watch_paths each time a build generates the pipeline, so a change to watch takes effect from the build it arrives with.

By default (build_type: auto), Canopy checks for a Dockerfile at your build_path first. If one exists, it’s built directly. Otherwise, Canopy falls back to Cloud Native Buildpacks - no Dockerfile required for common frameworks (Node.js, Python, Go, Ruby, Java, PHP, .NET, and more, via each buildpack’s own detection). Framework suggestions in the dashboard recognise the same set, plus Angular.

Set build_type explicitly if you’d rather not rely on that fallback:

canopy environments update <environment_id> --build-type dockerfile
canopy environments update <environment_id> --build-type buildpack
canopy environments update <environment_id> --build-type auto # back to the default

Under dockerfile, a missing Dockerfile fails the build outright instead of silently switching to buildpacks - useful if you want a missing Dockerfile to be a loud error, not a surprise builder switch.

When Canopy builds via buildpacks, it uses one of two Canopy-curated CNB builder images - never an arbitrary, customer-supplied builder:

  • heroku (default) - Heroku’s own maintained builder, broad language/framework coverage.
  • paketo - a Paketo Buildpacks builder, useful for stacks Heroku’s builder doesn’t cover as well (e.g. some JVM/.NET setups).
canopy environments update <environment_id> --cnb-builder paketo

cnb_buildpacks lets you pin an explicit, ordered list of buildpacks instead of relying on the builder’s own auto-detection - useful when detection picks the wrong buildpack for a polyglot repo, or you want to add a buildpack the default detection group doesn’t include (e.g. a team-registered third-party buildpack, once your team has one registered). Passing a non-empty list replaces the builder’s default detection group entirely - it doesn’t append to it - so include everything your build actually needs:

canopy environments update <environment_id> \
--cnb-buildpacks "heroku/nodejs,heroku/procfile"

Leave it empty (the default) to use cnb_builder’s own default detection order - the common case, and what “auto-detects your framework” means day to day.

Buildpack builds cache layers between builds by default (build_cache_enabled: true) so an unchanged dependency lockfile doesn’t get reinstalled on every deploy. Disable it if you’re debugging a build that seems to be serving stale cached layers, or want every deploy to build fully from scratch:

canopy environments update <environment_id> --build-cache-enabled false

This only affects buildpack builds - a Dockerfile build’s caching is whatever your Dockerfile/Docker’s own layer cache already does.

Checking an environment’s current build configuration

Section titled “Checking an environment’s current build configuration”
canopy environments show <environment_id>

prints build_type, cnb_builder, cnb_buildpacks, and build_cache_enabled alongside the rest of the environment’s build configuration.

An environment can require a human to approve a deploy after clone/build/test finish, before it actually goes live - useful for production environments where you want a final check. The build stays in an “awaiting approval” state until approved or rejected from the dashboard.

GET /teams/{team_id}/approvals lists every deploy of the team that is waiting (optionally ?project_id= and ?limit=, newest first). Each entry names the project and environment, the commit and its message, who asked and when, the commit that is live in that environment (live_commit), and the other environments of the project where the same commit already deployed successfully (verified_in), so a reviewer can see whether it ran in staging first. can_approve says whether the caller holds environment:approve_deploy; approving or rejecting is POST /environments/{environment_id}/builds/{build_id}/deploy-approval with { "action": "approve" | "reject" }. The dashboard shows the same list on the home page and on each project.

A failed, timed out or unhealthy deployment has an investigation page at /projects/{project}/environments/{environment}/pipelines/{build}/investigate, linked from the build page, the Incidents page, the Systems pages and the failed-build notification. It shows what failed, the chain from commit to runtime (a step Canopy did not record says “Not recorded”), the last deployment verified healthy and the commits since it, the log of the failing stage, and what depends on the environment.

Recovery uses the flows that already exist: the rollback confirmation that lists what will change, a retry where one is allowed, and the incident page. They are disabled, with the reason, when your role cannot deploy. Select log lines and choose “Attach to Ask Canopy” to give the assistant those lines (at most 200, secrets redacted by the server); suggestions are marked “Suggested, not verified”.