Deploys & the Staged Pipeline
Canopy runs a pipeline for each deploy. Its exact operations depend on the environment and its pipeline configuration. A Git-backed deploy checks out the selected commit, builds an image, runs any required approval and pre-deploy release tasks, then rolls out the services and checks their health. The build page reports the failing stage and reason.
The configured build_command is part of the image build configuration.
Configured build_steps run later as ordered release tasks, each in its own
task using the built image. They gate the deploy, but changes made in one
task’s container are not shared with the next task. The previously active
release remains in place if a release task fails.
Build numbers
Section titled “Build numbers”Every build has a number within its environment: 1, 2, 3 in creation order, never reused, with gaps possible if a build row is
removed. It is shown as #142 in the app and in canopy deploys list / canopy builds show, and it is the number field of
every build in the API (null on a build that has not been numbered yet). A preview environment numbers its own builds.
Wherever a route takes a build under an environment (/environments/{id}/builds/{build_id}... and
/environments/{id}/deployments/{build_id}), build_id may be the UUID or the number, 142 or #142 (URL-encoded as %23142).
A string of digits (no leading zero, at most nine digits) is always a number and a UUID never is, so the two cannot be mixed up.
A number the environment has no build for is a 404 with error_code: "build_number_not_found"; a number is never looked up
across environments. The CLI accepts 142 or '#142' (quote it: a shell reads an unquoted # as a comment) for the commands that
already need --env.
When a deploy counts as successful
Section titled “When a deploy counts as successful”A build becomes successful when its build pipeline and rollout complete:
the new web and worker services have converged and the runtime accepts the
release. A build held for human review is awaiting_approval. A terminal
rollout failure is reported on the build; a separate public HTTP check does
not hold the build open.
After a successful deploy, Canopy queues a bounded public verification for
web services whose public check is enabled and which have a domain. It probes
the configured health path up to six times over about 90 seconds and records
healthy or unhealthy separately from build status. No configured domain
or a disabled public check records unknown; an environment with no web
service records not_applicable. This verification never delays or fails a
deploy. See Environments & Services for service health
settings.
The deployment read (GET /environments/{id}/deployments/{build_id}) reports the verification as
status, attempts, checkedAt, probeTarget (the address probed) and detail (what the probe saw, for example
GET /health: no response within 30 s); the last two are null when the probe recorded none.
| Status | Meaning |
|---|---|
successful |
Built, deployed and healthy. |
failed |
A step failed or the pipeline could not complete. |
deploy_failed |
CI passed but the deploy did not become healthy. |
timed_out |
The pipeline or the deploy did not finish in its time budget. |
errored |
An internal Canopy error. |
cancelled / rejected |
Stopped by a person, or replaced by a newer build. |
canopy builds cancel <build_id> --env <environment_id> cancels one pending or running build: its unfinished steps and
Grove jobs are stopped and no other build is touched. Asking again for a build that is already cancelled answers 200 with
already_cancelled: true and changes nothing; a build that already succeeded or failed answers 409. The command reads the
environment’s slug itself for the confirmation described below.
A failed build carries a failure object (canopy builds show prints it, and
so do the API and MCP) saying where it stopped and whose problem it is:
stage:source,pipeline,artifact,capacity,admission,release_task,rollout,routing,tls,health_check,deploy_timeoutorinternal.kind:user_code,user_config,platform,infrastructure,capacityortimeout.detail: the specific reason, for exampleapp.example.test: hostname already claimedorHTTP 502.
Failure reports
Section titled “Failure reports”Next to failure, a failed build detail carries an error: the failure report that ended the build. Where failure
says which stage stopped, error says what happened to which node or generation, whose problem it is and whether trying
again can help:
reference: anfr_…value to quote to support. It identifies one failed attempt.category:source_fetch,configuration,build_step,test_failure,image_build,release_task,rollout,readiness,capacity,infrastructure,platform_defect,external_service,timeoutorunknown.owner:customer,canopy,externalorunknown.retryable:yes,noorunknown.operation: thenode_id,node_attempt,generationandrelease_idthe failure belongs to.messageanddetail(the exit code and an excerpt of the failing output) are present when the failure is yours. When Canopy owns the failure (owner: canopy) they arenullandpublic_messageis the explanation; quote the reference and Canopy staff can read the technical details.
error is null for a build that did not fail, for a failure Canopy later withdrew because the deploy converged, and
for builds that ran before failure reports existed. Every node in execution.nodes also has an error with the latest
report of that node, so a node that failed and then succeeded on a retry still names the attempt that failed
(outcome: recovered).
canopy builds failures (the API routes GET /environments/{environment_id}/builds/{build_id}/failures and
.../failures/{reference}) lists every report of a build, oldest first, including attempts that recovered. A build keeps
at most 32; dropped counts any refused past that. A reference that belongs to another build is a 404, the same as one
that does not exist. Output excerpts are masked with the build’s secrets and with common credential patterns, and are
kept for 90 days; the report itself stays with the build.
A build detail returns at most about 512 KiB of build_logs, the newest rows. When the log is larger,
build_logs_omitted says how many of the oldest rows were left out; it is 0 for a build whose log fits. The log stream
and the retained log are not affected.
A certificate or route that is still pending does not by itself fail a
rollout. A terminal runtime refusal or a timeout does. deploy_failed and
timed_out builds can be retried like a failed one. The later public
verification result is separate from these build statuses.
Build dependencies
Section titled “Build dependencies”An environment can depend on others when its build needs theirs finished first: an app that consumes a library built from the same
repository, for example. canopy environments depends-on <environment_id> --on <environment_id> makes the builds of the first wait
for the builds of the second for the same commit; --remove <environment_id> stops that, and
canopy environments dependencies <environment_id> (the API route GET /environments/{environment_id}/dependencies) shows both
directions. The App has the same under the environment’s settings, “Build dependencies”.
- A build waits while a build of an environment it depends on, for the same commit, is pending or running.
canopy builds showand the build page say what it is waiting for. - If one of those builds failed, errored, timed out or could not fetch its source, the waiting build ends as
failedwithout running and says which build failed, so it is never built against a broken dependency. A cancelled upstream build does not hold the build back or fail it. - Nothing to wait for means no waiting: if the dependency has no build of that commit, the build starts at once.
- An upstream build created more than two hours ago no longer holds a build back, so a stuck build cannot stall the queue.
- Dependencies are between environments of the same Organization, at most 20 per environment, and a cycle is refused. Builds of environments built from different repositories have different commits, so they never wait for each other.
Where a build’s time went
Section titled “Where a build’s time went”canopy builds show prints an efficiency section, and the build detail carries it as phase_timings and
execution.nodes[].measurements:
phase_timingslists how long Canopy spent generating the pipeline graph (phase_generate), planning it (phase_plan, with the time of each planning step such asexecution_keysandpolicies) and promoting the built image into the platform registry (phase_promote, with the blobs and bytes it copied and the blobs the registry already had). A build that did not do a phase has no entry for it.- Each node’s
measurementshold the scheduling, start-up and run time of its Pod and, for steps and image builds, the command time, peak memory and CPU the container used. A figure the machine did not report is absent, not zero.
Image-build figures appear for builds whose build worker image carries the reporting; older worker images report none.
Phases and counters of each job
Section titled “Phases and counters of each job”canopy builds steps <build_id> --env <environment_id> (the API route GET /environments/{environment_id}/builds/{build_id}/telemetry)
shows where each job of a build spent its time, as the build worker measured it:
- Phases: the steps BuildKit ran, the layers it pulled and unpacked and what it exported, each with how long it took and
whether it was cached.
--alllists every phase; by default the slowest twelve per job. - Counters: the worker samples the kernel’s cumulative counters every few seconds while it runs and the summary is the
difference between the first and last sample: CPU used and throttled, time spent waiting on I/O, CPU or memory (pressure
stall), disk and network bytes, peak memory.
--samplesalso lists how the counters changed between samples.
A figure the kernel does not give is shown as “not reported by this kernel” and is null in the API; it is not zero. A job
that reported nothing, because its worker or controller predates measurement, says so. --node <node_id> narrows to one job
and --json prints the jobs, phases and samples.
canopy builds timeline <build_id> --env <environment_id> draws each job on one time axis: how long it waited to start
(.) and how long it ran (#), so the longest wait or the job on the critical path is visible at a glance. A job from an
older API that sends no queue time is drawn from its start. --json prints the times.
canopy builds logs <build_id> --env <environment_id> prints the stored log of a build; --node <node_id> prints only the
log of one node of its pipeline and names the nodes there are when the one you gave does not exist.
Why a build is waiting
Section titled “Why a build is waiting”canopy builds activity <build_id> --env <environment_id> (the API route GET /environments/{environment_id}/builds/{build_id}/forest-activity)
lists, oldest first and in your local time, what Host and the Grove did for the build: jobs handed to the Grove, waits for
capacity with the scheduler’s own reason (for example Insufficient cpu), and pod events. A job that sits in running with
no matching line from the Grove is the sign that it never reached the Grove. Host fails such a job after 15 minutes
without any report from the Grove (lost: grove ... never reported on this job), instead of leaving the build waiting.
--json prints the lines.
Builds Canopy runs itself start one at a time: while one is in flight, the next stays pending until it finishes
(BUILD_MAX_LIVE_BUILDS, default 1), because a build Grove sized for one large build cannot schedule two.
Dependency images: reused, built or a fallback
Section titled “Dependency images: reused, built or a fallback”A step that declares dependencies gets an image with them installed once per lockfile (see the pipeline reference). The
build detail carries what became of each such step as dependency_resolutions, and canopy builds show prints one line each:
reused: a stored image, or one another step was building at the same time, was used.built: a new image was built for this step.fallback: no image was provided and the step installed its own dependencies. This is not the optimisation working. Thereasonis one ofUNSUPPORTED_STEP,MACHINE_BUILDS_NOT_ACCEPTED(the step runs on one of your own machines and its owner has not allowed image builds there),DRIVER_UNSUPPORTED,NO_BUILD,UNTRUSTED_RUN,NO_SOURCE,TREE_UNAVAILABLE,NO_PACKAGE_MANAGER,NO_LOCKFILE,IMAGE_CHECK_UNKNOWN,NO_ENVIRONMENT,IMAGE_BUILD_FAILED(with what the build said indetail) orERROR.
The same line is written at the top of the step’s log, and /metrics counts them as
canopy_dependency_resolutions_total{outcome,reason} and canopy_dependency_resolution_seconds_total.
Releases
Section titled “Releases”Every build also records a release: the image each service ran and the configuration it was built with.
A release is building while its build runs, built once it succeeds, and active once it is live. The release
that was live before it becomes superseded. A build that fails or is cancelled leaves a failed or
cancelled release and never changes which one is live.
GET /environments/{id}/releasesGET /environments/{id}/releases/{release_id}POST /environments/{id}/releases/{release_id}/rollbackEach release lists its services (image digest reference and a snapshot of the service settings) and the
service_builds that produced it; the build detail response carries the same release and service_builds.
Rolling back to a release creates a new rollback build, exactly like rolling back to the release’s build, and
records which release it restores. Only a release that was live can be restored (release_not_rollbackable),
not the active one (release_already_active), and only while the environment still has the services it had
then (release_service_set_mismatch).
Confirming a rollback or a cancel
Section titled “Confirming a rollback or a cancel”Rolling back (to a build, to a release, or to the last healthy build) and cancelling a build need the environment’s slug as confirm, in the JSON body or the query string, the way deleting a project needs its slug. Without it the answer is 409 with error_code: "confirmation_required", which names the slug to send:
{ "code": "conflict", "error_code": "confirmation_required", "title": "This will roll back environment \"production\". Pass confirm=production to go ahead.", "confirm_field": "confirm", "environment": { "id": "5f0e...", "slug": "production" }}The app, the CLI, the SDK (confirm argument) and the MCP tools send it for you.
Rolling back safely when the environment may have moved
Section titled “Rolling back safely when the environment may have moved”The app’s single Roll back button opens a picker: the last verified-healthy build, or any release that was live and has been replaced. It sends the live release it saw as expected_current_release_id, so a push that landed in between reloads the list instead of rolling back the wrong thing.
A rollback dialog shows the release that is live now and the one it will restore. Between the moment you looked and the moment you confirm, someone else (or a push) can deploy something new, and the rollback would then undo that instead. Every rollback endpoint therefore accepts an optional precondition:
POST /environments/{id}/releases/{release_id}/rollbackPOST /environments/{id}/builds/{build_id}/rollbackPOST /environments/{id}/builds/rollback (the newest build verified healthy)Send expected_current_release_id (the environment’s active release as you saw it, or null for none) and/or
expected_current_build_id (its current build), in the JSON body or the query string. If either no longer matches,
nothing is rolled back and the answer is 409 with error_code: "release_changed":
{ "code": "conflict", "error_code": "release_changed", "title": "The environment changed since you looked at it: its current release is no longer the one you expected.", "current_release_id": "5f0e...", "current_build_id": "9a21...", "retryable": false}Show the new state, let the user decide again, and retry with the new ids. The check is made again when the rollback
build is created, under a lock on the environment, so two rollbacks carrying the same expectation cannot both be
accepted. Omit both fields to keep the previous behaviour. A value that is not a UUID is a 400.
Several services: one release, all or nothing
Section titled “Several services: one release, all or nothing”When an environment has several services, each is built from its own folder and the release goes live only when
every web and worker service has converged on it. If one fails, the services already updated are put back on the
previous release and the release is reported reverted (or failed when there was nothing to go back to), with the
service that failed and where. While a release is rolling out, services can briefly serve different releases:
Kubernetes cannot switch several deployments at one instant. Jobs never block a release.
Rolling back to a release of a multi-service environment redeploys the images that release was made of, all at once, without building anything. If the build worker restarts while a release is rolling out, the release is settled within a couple of minutes from what the services report.
build_command vs build_steps
Section titled “build_command vs build_steps”canopy build-steps add "npm ci"canopy build-steps add "npm test"canopy build-steps list --env <environment_id>build_steps run in order as separate pre-deploy tasks in the built image.
Use them for checks or commands that need the image and its runtime
configuration; they do not share filesystem changes with later steps. For
example, if the image includes a pre-deploy check script:
canopy build-steps add "node scripts/predeploy-check.mjs" --env <environment_id>canopy build-steps list --env <environment_id>Use the image builder or Dockerfile to install dependencies and produce
build outputs. A single --build-command on environments update configures
the build command; it is not a replacement for the ordered release tasks.
Monorepos and build_path
Section titled “Monorepos and build_path”Set --build-path (on environments create/environments update) to
build from a subdirectory instead of the repo root - this is also how
multi-service environments let one repo’s frontend and
backend deploy independently.
A push builds an environment only when it changed the environment’s build_path. Canopy reads that from the file lists in
the push payload, except when the payload cannot be trusted: it was cut short (GitHub and GitLab list at most 20 commits), or
it holds a merge commit, whose file list is taken against its first parent and not against the branch head the push moved
from. In those cases Canopy compares the branch’s previous head with its new head itself, so pulling before you push does not
rebuild the environments the other side of the merge had already built. When that comparison cannot be made (a new branch, a
head the provider cannot serve) the environment builds, because skipping a real change is worse than one extra build.
A pipeline can name more paths than the build path with watch in its canopy.pipeline.ts (a shared SDK, a script, committed
baselines the build reads; see Canopy Pipelines). The environment
then also builds when a push touches one of them, and a skipped push (skipped_path in the webhook deliveries) lists them as
watch_paths next to the build_path. Host stores the list on the environment as watch_paths each time a build generates
the pipeline, so a change to watch takes effect from the build it arrives with.
Build methods: Dockerfile vs. buildpacks
Section titled “Build methods: Dockerfile vs. buildpacks”By default (build_type: auto), Canopy checks for a Dockerfile at your
build_path first. If one exists, it’s built directly. Otherwise, Canopy
falls back to Cloud Native Buildpacks - no
Dockerfile required for common frameworks (Node.js, Python, Go, Ruby,
Java, PHP, .NET, and more, via each buildpack’s own detection). Framework suggestions in the dashboard
recognise the same set, plus Angular.
Set build_type explicitly if you’d rather not rely on that fallback:
canopy environments update <environment_id> --build-type dockerfilecanopy environments update <environment_id> --build-type buildpackcanopy environments update <environment_id> --build-type auto # back to the defaultUnder dockerfile, a missing Dockerfile fails the build outright
instead of silently switching to buildpacks - useful if you want a
missing Dockerfile to be a loud error, not a surprise builder switch.
cnb_builder and cnb_buildpacks
Section titled “cnb_builder and cnb_buildpacks”When Canopy builds via buildpacks, it uses one of two Canopy-curated CNB builder images - never an arbitrary, customer-supplied builder:
heroku(default) - Heroku’s own maintained builder, broad language/framework coverage.paketo- a Paketo Buildpacks builder, useful for stacks Heroku’s builder doesn’t cover as well (e.g. some JVM/.NET setups).
canopy environments update <environment_id> --cnb-builder paketocnb_buildpacks lets you pin an explicit, ordered list of buildpacks
instead of relying on the builder’s own auto-detection - useful when
detection picks the wrong buildpack for a polyglot repo, or you want to
add a buildpack the default detection group doesn’t include (e.g. a
team-registered third-party buildpack, once your team has one
registered). Passing a non-empty list
replaces the builder’s default detection group entirely - it doesn’t
append to it - so include everything your build actually needs:
canopy environments update <environment_id> \ --cnb-buildpacks "heroku/nodejs,heroku/procfile"Leave it empty (the default) to use cnb_builder’s own default
detection order - the common case, and what “auto-detects your
framework” means day to day.
build_cache_enabled
Section titled “build_cache_enabled”Buildpack builds cache layers between builds by default
(build_cache_enabled: true) so an unchanged dependency lockfile doesn’t
get reinstalled on every deploy. Disable it if you’re debugging a build
that seems to be serving stale cached layers, or want every deploy to
build fully from scratch:
canopy environments update <environment_id> --build-cache-enabled falseThis only affects buildpack builds - a Dockerfile build’s caching is
whatever your Dockerfile/Docker’s own layer cache already does.
Checking an environment’s current build configuration
Section titled “Checking an environment’s current build configuration”canopy environments show <environment_id>prints build_type, cnb_builder, cnb_buildpacks, and
build_cache_enabled alongside the rest of the environment’s build
configuration.
Manual deploy approval
Section titled “Manual deploy approval”An environment can require a human to approve a deploy after clone/build/test finish, before it actually goes live - useful for production environments where you want a final check. The build stays in an “awaiting approval” state until approved or rejected from the dashboard.
GET /teams/{team_id}/approvals lists every deploy of the team that is waiting (optionally ?project_id= and
?limit=, newest first). Each entry names the project and environment, the commit and its message, who asked and when,
the commit that is live in that environment (live_commit), and the other environments of the project where the same
commit already deployed successfully (verified_in), so a reviewer can see whether it ran in staging first.
can_approve says whether the caller holds environment:approve_deploy; approving or rejecting is
POST /environments/{environment_id}/builds/{build_id}/deploy-approval with { "action": "approve" | "reject" }. The
dashboard shows the same list on the home page and on each project.
Investigating a failed deployment
Section titled “Investigating a failed deployment”A failed, timed out or unhealthy deployment has an investigation page at /projects/{project}/environments/{environment}/pipelines/{build}/investigate, linked from the build page, the Incidents page, the Systems pages and the failed-build notification. It shows what failed, the chain from commit to runtime (a step Canopy did not record says “Not recorded”), the last deployment verified healthy and the commits since it, the log of the failing stage, and what depends on the environment.
Recovery uses the flows that already exist: the rollback confirmation that lists what will change, a retry where one is allowed, and the incident page. They are disabled, with the reason, when your role cannot deploy. Select log lines and choose “Attach to Ask Canopy” to give the assistant those lines (at most 200, secrets redacted by the server); suggestions are marked “Suggested, not verified”.