Skip to content

fix: drop docker/docker and bump buildkit to v0.31.2 via the kaniko fork - #535

Open
stirby wants to merge 3 commits into
mainfrom
stirby/kaniko-moby
Open

stirby wants to merge 3 commits into
mainfrom
stirby/kaniko-moby

Conversation

@stirby

@stirby stirby commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Pins the kaniko fork to coder/kaniko#38, which drops github.com/docker/docker and the chrismellard ACR helper and bumps buildkit to v0.31.2. This fixes the remaining reachable advisories in envbuilder.

Generated by Coder Agents on behalf of @stirby.

Depends on coder/kaniko#38. The replace points at 61a5b7da on that PR. Later commits there only change CI and integration fixtures, not Go code. Re-pin to the merge commit once #38 lands.

Test template on dogfood, which builds envbuilder from this branch: https://dogfood.cdr.dev/templates/coder/kirby-envbuilder-rollup

Advisories

govulncheck ./... (reachable):

main (a5b7329) this branch
GO-2026-4858, GO-2026-4859, GO-2026-6255 (moby/buildkit) affected fixed (v0.31.2)
GO-2026-4883, GO-2026-4887 (docker/docker) affected fixed (removed from kaniko)
GO-2026-6253 (moby/go-archive) affected fixed (v0.3.3)
GO-2026-6225 (chrismellard/docker-credential-acr-env) affected fixed (ported with anchored host check)
Total 7 0
  • Binary scan: govulncheck -mode=binary on cmd/envbuilder reports only GO-2026-6443 (grpc). grpc is intentionally left alone in this PR.
  • What's not in the binary: docker/docker, the moby/moby monolith and chrismellard.
  • What module-level scanners will still flag: docker/docker and moby/moby stay in go.mod as the Docker client for ./integration tests only. Also x/crypto openpgp and dhcp findings, which are unreachable.

Changes

  • kaniko fork re-pinned: buildkit v0.16.0 → v0.31.2, go-containerregistry v0.20.7 → v0.21.7, go-archive v0.1.0 → v0.3.3. The rest are tidy fallout.
  • devcontainer.UserFromDockerfile: passes a buildkit linter to ParseInstructionWithLinter. Without one, buildkit v0.31 panics on Dockerfiles with # check= directives. Test: TestUserFromDockerfile_BuildArgs/LintCheckComment.
  • features.Extract: now closes the feature layer reader. go-containerregistry v0.21 holds a pull slot until the reader is closed.

Behavior changes

  • Unsupported Dockerfile flags: kaniko now fails COPY/ADD --exclude, COPY --parents, ADD --unpack, RUN --security=insecure and RUN --device with "not supported by kaniko". 1.3.0 also failed on these, as unknown flags.
  • *.local registries: go-containerregistry v0.21 no longer falls back to plain HTTP for them. Only localhost, *.localhost and loopback IPs fall back. Plain-HTTP in-cluster registries (for example registry.ns.svc.cluster.local:5000) need ENVBUILDER_INSECURE=true.
  • Private IP literals: go-containerregistry v0.21 rejects token realms, blob redirects and upload locations pointing at a different private IP literal.
  • Registry retries: backoff is now 1s/3s/9s, and 429 responses are retried, so a failing registry takes longer to error out.

Validation

  • make test passes (all 36 integration tests, 0 skipped), as do make lint, make fmt and make gen (no diff).
  • Local e2e with an image built from this commit:
    • The coder/coder main devcontainer (64-layer base image) builds to INIT. An earlier pin of this branch deadlocked at layer 5 (go-containerregistry pull limiter, fixed in kaniko#38).
    • A multi-stage Dockerfile with FROM <stage> and COPY --from=golang:1.25-bookworm builds. The earlier pin deadlocked in "Storing source image".
    • A # check= Dockerfile builds. COPY --exclude fails with the new error, and a plain Dockerfile builds.
  • Dogfood: workspace envbuilder-rollup-main on the test template builds coder/coder main. The agent is ready, /.envbuilder/built exists, and code-server returns healthy. The binary reports vcs.revision=9e64f32, coder/kaniko 61a5b7da and go-containerregistry v0.21.7.
  • kaniko#38 CI: all green, including the k3s integration tests.
  • Terraform provider: chore: bump envbuilder and kaniko to drop docker/docker and update buildkit terraform-provider-envbuilder#138 mirrors this pin (it copies envbuilder's replace directives) and its acceptance tests pass.
Decision log
  • docker/docker: it has no fixed release. The moby fixes exist only in github.com/moby/moby/v2 betas, so kaniko ports the few helpers it used instead of upgrading.
  • chrismellard: the module has no fix and is unmaintained, so it was ported into kaniko with an anchored hostname regex.
  • go-containerregistry v0.21.7, not v0.21.6: v0.21.6 tarball.Write leaks pull slots and hangs multi-stage builds. A subagent review of the v0.19 → v0.21 diff found it, and it was reproduced locally.
  • .local HTTP fallback documented, not shimmed: it is an upstream security hardening. ENVBUILDER_INSECURE covers kaniko pulls, pushes and the cache repo. It does not cover envbuilder's own devcontainer image config and feature fetches, which never honored it.
  • Out of scope: grpc (GO-2026-6443) and moving ./integration off docker/docker.

stirby added 2 commits October 8, 2026 17:08
Pins coder/kaniko to 8e293b69, which removes github.com/docker/docker,
bumps buildkit to v0.31.2 and go-archive to v0.3.3, and vendors a fixed
ACR credential helper. Fixes GO-2026-4858, GO-2026-4859, GO-2026-6255,
GO-2026-6253, GO-2026-6225, and removes docker/docker (GO-2026-4883,
GO-2026-4887) from the envbuilder binary.

Parse Dockerfiles with an initialized linter: buildkit v0.31 applies
"# check=" comments to the linter and panicked with a nil one.
…v0.21.7

go-containerregistry v0.21 holds a pull limiter slot per open remote blob
reader. The previous kaniko pin deadlocked extracting base images with more
than four layers, and v0.21.6 tarball.Write hangs the same way.

Also close the devcontainer feature layer reader.
Comment thread go.mod
// There are a few options we need added to Kaniko!
// See: https://github.com/GoogleContainerTools/kaniko/compare/main...coder:kaniko:main
replace github.com/GoogleContainerTools/kaniko => github.com/coder/kaniko v0.0.0-20251107135632-b20ff58093d8
replace github.com/GoogleContainerTools/kaniko => github.com/coder/kaniko v0.0.0-20261008191411-61a5b7da7ea3

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pin points at an unmerged commit(v0.0.0-20261008191411-61a5b7da7ea3) - needs re-pined after merged and make sure coder/kaniko#38 is merged first 

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. coder/kaniko#38 is still open, so I've left the pin as is for now. Once #38 merges, I'll re-pin the replace to the merge commit on kaniko main and rerun the checks before this PR merges. This thread can stay open until then as a merge blocker.

Generated by Coder Agents on behalf of @stirby.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 [CRF-2] coder/kaniko#38 merged as a51d65b5 on 2026-10-14, and the commits after the pinned 61a5b7da change Go code, so the re-pin is not a no-op. (Ryosuke, Mafu-san, Pariston, Mafuuu, Gon, Leorio)

cd77cdb9 and 8cd8fc68 change pkg/creds/acr/token.go to request the service principal token for https://containerregistry.azure.net instead of settings.Environment.ResourceManagerEndpoint, and b7fcaf3d raises kaniko's go directive from 1.25.9 to 1.26.9, which forces envbuilder's go 1.26.5 up. The description's statement that later commits "only change CI and integration fixtures, not Go code" is incorrect, and none of the listed validation covered the ACR change. Re-pin to a51d65b5 in this PR, run go mod tidy, and update the description and the coder/terraform-provider-envbuilder#138 pin to match.

🤖

@Lisa-Fiander Lisa-Fiander left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few small comments

Comment thread go.mod
@@ -6,7 +6,7 @@ toolchain go1.26.6

// There are a few options we need added to Kaniko!

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not caused by this PR, but advisories published yesterday now flag toolchain go1.26.6 (12 reachable stdlib advisories, fixed in 1.26.9) and golang.org/x/net v0.58.0 (fixed in v0.60.0). Worth bumping both here or in a quick follow-up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bumped in 77452e6: toolchain go1.26.9 and golang.org/x/net v0.60.0. x/net v0.60.0 also requires newer x/crypto (v0.57.0), x/sys, x/term, x/text, x/sync and x/mod, so go mod tidy moved those too.

Checks: govulncheck ./... reports 0 reachable advisories. make test (integration included), make lint/go, make fmt and make gen all pass with no diff.

Generated by Coder Agents on behalf of @stirby.

@BobbyHo

BobbyHo commented Oct 15, 2026

Copy link
Copy Markdown

/coder-agents-review Please limit the review to P0, P1, and P2 issues only.

@coder-agents-review

coder-agents-review Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Chat: Review posted | View chat
Requested: 2026-10-15 01:28 UTC by @BobbyHo

Review history
  • R1 (2026-10-15): 11 reviewers, 2 P2, COMMENT. Review

deep-review v0.13.0 | Round 1 | a5b7329..77452e6

Last posted: Round 1, 2 findings (2 P2), COMMENT. Review

Finding inventory

Finding inventory: PR #535

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P2 Open go.mod:30 gcr v0.21.7 drops *.local HTTP fallback; envbuilder's own image/feature fetches (3 call sites) ignore ENVBUILDER_INSECURE, no workaround R1 Netero, Mafuuu, Razor Yes
CRF-2 P2 Open go.mod:9 kaniko#38 merged as a51d65b5; commits after the 61a5b7da pin change ACR token code and kaniko's go directive, contradicting the description R1 Ryosuke P2, Mafu-san P2, Pariston P3, Mafuuu P3, Gon P3, Leorio P3 Yes
CRF-3 P4 Dropped by orchestrator (downgraded from P2, comment is accurate and causes no defect; requester limited review to P0-P2) devcontainer/devcontainer.go:359 Linter comment states the panic condition broadly R1 Gon No
CRF-4 P4 Dropped by orchestrator (downgraded from P2, either # check= line alone keeps the test red; requester limited review to P0-P2) devcontainer/devcontainer_test.go:286 Test fixture # syntax note line unexplained R1 Gon P2, Bisky P4 No
CRF-5 P4 Dropped by orchestrator (downgraded from P2, comment lines unchanged by the PR and already stale; requester limited review to P0-P2) go.mod:7 Fork comment no longer describes why the fork exists R1 Gon No
CRF-6 P3 Dropped by orchestrator (requester limited review to P0-P2) docs/development.md:37 Doc still says kaniko fork pins docker/docker R1 Gon No
CRF-7 P3 Dropped by orchestrator (requester limited review to P0-P2) options/options.go:389 ENVBUILDER_INSECURE help text says TLS only, not plain HTTP R1 Leorio No
CRF-8 P3 Dropped by orchestrator (requester limited review to P0-P2) go.mod:5 Toolchain/x/net bump missing from description; dogfood ran prior commit R1 Mafu-san P3, Leorio P4 No
CRF-9 P4 Dropped by orchestrator (requester limited review to P0-P2) devcontainer/features/features.go:39 layerReader hoisted; nil pairing with tarLayer implicit R1 Ryosuke No
CRF-10 Nit Dropped by orchestrator (requester limited review to P0-P2) devcontainer/devcontainer_test.go:270 LintCheckComment placed under BuildArgs test R1 Mafuuu No
CRF-11 P4 Dropped by orchestrator (requester limited review to P0-P2) go.mod:9 Description "1.3.0" does not name the product R1 Leorio No
CRF-12 Note Dropped by orchestrator (requester limited review to P0-P2) devcontainer/features/features.go:58 Pull-slot rationale cannot cause a hang on this path (per-call limiter) R1 Pariston, Bisky, Mafu-san No
CRF-13 OOS Out of scope (body) devcontainer/features/features.go:68 Feature tar entry names not contained; ../ entries write outside the feature dir R1 Kurapika No
CRF-14 OOS Out of scope (body) devcontainer/devcontainer.go:487 envbuilder's own remote.Image calls ignore ENVBUILDER_INSECURE for TLS verification too R1 Netero No
CRF-15 OOS Out of scope (body) buildkit instructions/parse.go:67 Upstream ParseInstruction passes nil linter and panics on # check= comments R1 Ryosuke No

Contested and acknowledged

Round log

Round 1

Round type: round 1, full panel. Netero first pass: 1 P2 (CRF-1), no P0, so panel ran. Law not run (effective additions 56).
Panel: ging-go, ryosuke, kurapika, killua (specialists); pariston, mafuuu (core); bisky, gon, leorio, mafu-san (pool); razor (wildcard). Multi-domain target used because the PR is a security advisory fix.
Requester (PR comment IC_kwDOJPSNMM8AAAABcH39Kw) asked to limit the review to P0-P2; P3 and below recorded here, not posted.
Downgrade gate:

  • CRF-3 keep-argument: the broad wording already misled the PR description. Downgraded because the comment's mechanism is accurate (Leorio, Razor verified) and wording causes no behavior defect.
  • CRF-4 keep-argument: a maintainer could delete # syntax note and weaken the test. Downgraded because Bisky verified the # check= before RUN alone still panics, so the test stays red.
  • CRF-5 keep-argument: the comment sits on the replace line this PR changes. Downgraded because the comment text is unchanged and was already inaccurate before the PR.
    CRF-2 severity: P2 per Ryosuke and Mafu-san (higher wins); verified locally with git log 61a5b7da..a51d65b5 on coder/kaniko main (merge of Deal with devcontainer.json "liberties" #38, 2026-10-14) touching pkg/creds/acr/token.go and go.mod (go 1.25.9 -> 1.26.9).
    CRF-1: Mafuuu and Razor added a third call site, envbuilder.go:500 (SkipRebuild path).
    Reviewed against a5b7329..77452e6.
About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR re-pins the kaniko fork to drop docker/docker and bump buildkit and go-containerregistry, and adapts two envbuilder call sites to the new versions. 2 P2 findings. As requested in this thread, only P0-P2 findings are posted.

Out of scope (needs a ticket or explicit acceptance by a human):

  • devcontainer/features/features.go:68: filepath.Join(directory, header.Name) does not reject ../ tar entries, and the default filesystem is osfs.New("/"), so a feature image can write files anywhere on the build filesystem (inferred from code, not run).
  • devcontainer/devcontainer.go:487: envbuilder's own remote.Image calls also ignore ENVBUILDER_INSECURE for TLS verification, so base images and features on a self-signed registry fail regardless of the flag.
  • buildkit v0.31.2 frontend/dockerfile/instructions/parse.go:67: ParseInstruction passes a nil linter and panics on any # check= instruction comment, so any other caller of it crashes on such Dockerfiles.

🤖 This review was automatically generated with Coder Agents.

Comment thread go.mod
github.com/go-git/go-git/v5 v5.19.2
github.com/google/go-cmp v0.7.0
github.com/google/go-containerregistry v0.20.7
github.com/google/go-containerregistry v0.21.7

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 [CRF-1] go-containerregistry v0.21.7 no longer falls back to plain HTTP for *.local registries, and ENVBUILDER_INSECURE=true does not restore it for envbuilder's own image and feature fetches. (Netero, Mafuuu, Razor)

reLocal in pkg/name/registry.go:28 changed from .*\.local(?:host)? to .*\.localhost. UserFromImage (devcontainer/devcontainer.go:487), extractFromImage (devcontainer/features/features.go:29) and the SkipRebuild path (envbuilder.go:500) call remote.Image on references parsed without name.Insecure, so a devcontainer whose base image or features are on registry.ns.svc.cluster.local:5000 now fails in Compile or on restart, with or without the flag (read from both module versions, not run against a live registry). Parse these three references with name.Insecure when opts.Insecure is set, so the workaround in the description covers them.

🤖

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants