Repository navigation
Conversation
…ds and werf charts
Four gaps of the deployment graph, closed together.
Images named from a default value keep their Image; the edge says so.
A Dockerfile FROM / COPY --from filled by an ARG default (an ARG chain
names each ARG) carries default_of, as do compose `${X:-d}` images and
`docker build -t ${IMAGE:-x}` tags. An override found in the repository
adds its own edge: compose build.args and --build-arg make the Image the
build names DEPEND_ON the base they choose {override_of}; compose's .env
replacing an inline default marks the effective edge override_of and keeps
the default as {ref: image_default}, an image the service does not run.
Helm template images render (helm_template.c): a bounded Go
text/template + Sprig evaluator over the vendored gotmpl grammar, with
values read the way Helm reads them (go-yaml v2 scalars, float64 numbers,
Helm's coalesce). Each templated container image renders from its chart
family's defaults -- include / define helpers, with / range / if, subchart
values under their alias, global -- and the object USES it {default_of:
the values keys read}; the chart DEPLOYS it. A value that does not render
(a .Release field, lookup, a missing define, a field of a missing map, a
function not modelled) names no Image; a rendered value is checked against
the image reference grammar first. Releases render their charts with their
values: a Flux HelmRelease's spec.values, a kustomize helmCharts entry's
valuesFile + valuesInline (a chart in chartHome now links locally), a
Terraform helm_release's values + set (literals only; set_sensitive is
never read) -- each image that differs is the release's own DEPLOYS
{override_of}. A release of a chart outside the repository names images by
convention only where its values' `repository` names a registry host
{inferred: values}; a host-less one names none.
A chart DEPLOYS the CRDs of its crds/ {phase: crds}. A werf `.helm` dir
beside werf.yaml is a chart named after the werf project.
Incremental runs: a werf.yaml edit and a werf chart's values / templates
decline like a Chart.yaml chart's; a release's values file (a kustomize
valuesFile, a Terraform file()) declines too -- Terraform's now IMPORTS
{local_name: values} it, like kustomize's valuesFile.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…ng release images Review round 1 of #2677. Every case ends in an explicit unknown, never in a hang, an OOM, a partial or a wrong image. Bounds (helm_template.c): - A per-render step and output budget (10,000,000 steps, 64 MiB): a range over a huge int, a huge indent, defines that include each other in a fan stop the render; every probe of the template is UNKNOWN, also one captured before, and the trip is logged (helm.render.budget). Counted, never timed. On the 20 infra corpora no render reaches it. - go-yaml v2's alias limit: values whose decodes are mostly alias expansion (a billion laughs) do not parse, as in Helm. - --set: an index past strvals' MaxIndex (65536) or a key longer than the parser holds is no value (the release is unknown), never a padded list or a split key. Reads (pass_k8s.c, pass_terraform.c): a values file, Chart.yaml, kustomize valuesFile or Terraform file() named by path is read only when it is a discovered file of the repository -- never through a symlink out of it -- with the size cap and tree-sitter padding of every other read. A chart file that is there but not read makes the chart's values unknown, not empty. Wrong images: - Terraform number sets reach the chart as Terraform's exact decimal (1.10 -> "1.1"), not %.17g. - A release with a value source not read -- Flux valuesFrom, valuesFiles, postRenderers; kustomize additionalValuesFiles, a valuesMerge other than override, a valuesFile not read; Terraform dynamic "set" -- has no image of its own. - An aliased subchart renders with .Chart.Name its alias. - An UNKNOWN if / with / range makes what its bodies assign UNKNOWN and leaves the loop a break in them would; set / unset / merge poison the map they change (merge: the maps within too). A change that may reach values another part holds taints every later capture; one of a new value (fromYaml, deepCopy, an empty dict, a part of one) does not. Marks and secrets: - The chart edge reads default_of as JSON (list paths kept), builds its properties to size, and writes no edge whose mark it could not carry whole. - Build arguments with a secret (by name or value) are never kept; build edge properties and values-path marks are never cut. Tests: one per bound and per wrong-image case, each RED on d21949e by its assertion; a Helm chart joins the Kubernetes complexity fixture. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…change Review round 2 of #2677 (N1-N12). Every case ends in an explicit unknown, never in an OOM, a quadratic stall or a wrong image. Bounds (helm_template.c, pass_k8s.c): - N1: capture arrays grow by doubling, so n captures cost memory linear in n. Arena growth since the render began counts against a third budget (256 MiB) beside steps and output; a trip is logged with its reason. - N10: poisoned maps are a pointer hash set (merged values a path list), so a loop poisoning a new dict per iteration keeps reads O(1). Release images are deduplicated by hash. Wrong images (helm_template.c): - N2: a set / merge in a body an UNKNOWN condition skips is judged by its target whatever it is -- a call's result like `index .Values "image"` is evaluated in the frame: own, or a known map (poisoned), else it taints. - N3: putting a value another part holds into an own map revokes own; a part of a poisoned map is never own. - N4: a run-time set whose target is an UNKNOWN call result taints. - N5: an UNKNOWN break / continue spoils the rest of the loop body around it; a continue is a continue, not a break. - N6: the scan notes redefinitions and assignments, carries what it saw across the actions of one branch, and never judges a variable the part rebinds by its value in the frame. - N11: a define's "may not change" that leaned on a define still being scanned (mutual includes) is not kept. - N12: an include not run because its argument is poisoned or UNKNOWN spoils that argument (poisoned deep, or a taint). Releases and charts (pass_k8s.c, pass_terraform.c): - N8: an empty values.yaml is values {} (Helm's), not a file not read. - N9: a Terraform set_sensitive whose name overlaps a key an image reads, a Terraform postrender, or a kustomization with its own transformers (images, patches, replacements, transformers, components) leaves the release without an image of its own. Tests (N7): the symlink probes are POSIX-only (SKIP_PLATFORM on Windows). One test per item, RED on 4cea54f by its assertion; N10's is a correctness control, its cost measured (linear vs quadratic) outside the suite. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
… render memory Review round 3 of #2677. The user decided on a rule sound by construction: a change of maps the render does not run exactly leaves every later image of that render UNKNOWN, with no exemption. Changes of maps (helm_template.c): - Under UNKNOWN control -- a skipped if / with / range body, the rest of a loop an UNKNOWN break / continue may leave, an include or template not run, a tpl not run -- any set / unset / merge (the must* forms too), any tpl, and any include / template of a name not known, of no define or template file of the chart, or of one that may change maps, leaves the images after it UNKNOWN. - An executed set / merge does too unless its target is a known map that is poisoned exactly; a merge also for a target already poisoned, the .Values root, or a map not known within it. The own provenance and the freshness scan are gone. - .Template.BasePath is modelled as Helm sets it: <chart path>/templates, the top chart's name, then /charts/<alias or name> per subchart (pass_k8s.c). An include naming one of the chart's template files by it (the checksum idiom) is scanned with the same rule; a file reached again while scanned, or no such file, may change maps. Bounds (helm_template.c): - A capture's keys are a hash set and its mark is built in one pass (one sort for the prefix check), its steps charged. - Each render works in an arena of its own; the caller keeps its results only. - The trip reason is recorded where a budget trips: an output trip is logged as output. Releases (pass_terraform.c, pass_k8s.c): - Any set_sensitive leaves the release without an image of its own (the name-overlap check is gone), and so does a postrender written as a dynamic block or an attribute. Cost on the 20 infra corpora against 5c5964e: 20 Image edges lost, 0 added, no budget trip. 5 are changes the rule rightly cannot follow (a second mustMerge into a poisoned values map; a skipped part that may change maps), 15 come from checksum-included files that hold a tpl. Two controls of rounds 1-3 now read UNKNOWN; their tests say so. Tests: one per item, each failing on 5c5964e by its assertion (the BasePath ones also on the coarse rule without it); the wrong-image scenarios of rounds 2 and 3 stay as tests. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This closes the four gaps recorded in #2663's "Found, not changed here" and #2654's "Default values still name images".
include/definehelpers and release values, by a bounded Go template evaluator (src/pipeline/helm_template.c).crds/..helm/dir is a chart.No new node label or edge type is added. The image identity rules of #2654 stay: no registry host is ever invented.
1. Default-valued images (
pass_docker.c,pass_ci.c,pass_shell.c)FROM/COPY --fromfilled by anARGdefaultdefault_of: ["GO","BASE"], every ARG of a chainimage: ${X:-d},.envnot setting Xdefault_of: ["X"]docker build -t ${IMAGE:-x}(also throughIMAGE=${IMAGE:-x})default_of: ["IMAGE"]build.args,docker build --build-argoverride_of: ["GO"],dockerfile,tool,file,build_line.envreplacing an inline defaultoverride_of,override_source: ".env"; the default kept as{ref: image_default}(not run, so the env join and hubs ignore it)${X:-d}with noARG Xis not marked, because no build argument can set it.2. Helm template images (
helm_template.c,pass_k8s.c,pass_terraform.c)Values are read the way Helm reads them:
%v(1.10→1.1,20231010→2.023101e+07);nulldeletes a key;--settyping forset.Templates render with Go text/template semantics:
{{-/-}}trimming;if/else if/with/range(map keys sorted) /break;define/template/block/include/tpl;and/or;Anything not modelled is UNKNOWN, and an image that reads an UNKNOWN names no Image:
.Release.*,.Capabilities,lookup, a missing define, a field of a missing map. A rendered value is checked against the image reference grammar before it names an Image.What the graph gets:
global; a dependencyconditionthat is false disables the subchart.{default_of: <values keys read>}.<release> -DEPLOYS {via, ref: image, image, override_of}-> Image. Release sources:spec.values;helmChartsvaluesFile+valuesInline. A chart withoutrepo:now links to the chart incharts/instead of an external node;helm_releasevalues = [...](heredoc,file(),yamlencode) thenset(block or provider-v3 list). Literals only: an interpolation orset_listleaves the release unread.set_sensitiveis never read.repository(+tag/digest/registry), and only when the reference names a registry host. That gives{inferred: values, values_key}; a host-less repository names none.3. and 4.
crds/and werfcrds/:Chart -DEPLOYS {via: chart, phase: crds}->each CRD of<chart>/crds/.werf.yamlbeside a.helm/withoutChart.yamlmakes a Chart named after the werf project, and the dir links like any chart: objects, values, includes, images.deploy.helmChartDiris not followed.Incremental runs (
pipeline_incremental.c)These edits now decline to the full rebuild, like their Chart.yaml-chart equivalents:
werf.yamledit (helm_chart_changed);valuesFileor a Terraformfile()(helm_release_values_changed). Terraform's values file now getsIMPORTS {local_name: values}, like kustomize'svaluesFile.No index format bump: the change only adds edges and properties.
Proof on real input
All 20 pinned infra corpora, base
9f85f104against this branch.Oracle:
helm templatewith Helm 4.3.1, local only,--kube-version 1.30.0, on every root chart whose dependencies are in the repository (84 charts).include({{ include "x.podSpec" . | nindent }}). There is noimage:key in the template to render; recorded below.helm_libis a symlink not in the corpus copy, and two charts are required-value or YAML errors.What the graph gains:
{default_of}{default_of}{inferred: values}(home-ops 28){default_of}(ARG defaults, chains){override_of}(--build-arg, deckhouse){phase: crds}includeCALLS edges are now drawn, and the similarity pass moves accordingly (+14 / −1 SEMANTICALLY_RELATED).helm template -f / --setoutput.Tests
All of these are RED on
9f85f104by their assertions, then green:probe_docker_default_images_marked_and_overridesprobe_helm_template_images_render_with_defaults: a bitnami-stylecommon.images.imagehelper,with, the if/else digest branch, a subchart alias with overlay and global;.Release.Name, a missing define and a field of a missing map render nothing.probe_helm_release_images: HelmRelease local and external, kustomize local, Terraformset.probe_helm_chart_crds_and_werf_chartincremental_werf_values_edit_keeps_chart_edgeandincremental_release_values_edit_rebuilds. With only their decline rule reverted they are RED at the edit.probe_helm_chart_deploys_its_objectsnow expects its templatednginximage (host-less, kept as written).helm_values_follow_helm_semanticsunit-tests the values rules. It tests the new module directly, so it has no RED-on-base run.Verification
lint-no-suppressandlint-memory-coreare clean.Found, not changed here
include(5 of 110 oracle images): the container'simage:key is in the define, not the template.helm.values/valuesObject, FluxvaluesFrom/chart.spec.valuesFiles, kustomizevaluesMerge/additionalValuesFiles/helmGlobals.chartHome, werfdeploy.helmChartDir, subcharttags.docker run/docker pullin scripts, Makefile?=, skaffoldbuildArgs, compose multi-file merges,docker-bake.hcl,--targetstage pruning.image:gets no override edge: its Service source would go stale on a Dockerfile-only edit.crds/, but files there containing{{are still skeletonized before parsing.Review round 1: requested changes
Every case below ends in an explicit unknown (no image), never in a hang, an OOM, a partial or a wrong image.
Bounds (
helm_template.c)indentor defines that include each other in a fan stop the render. Every probe of that template is UNKNOWN, including one captured before the trip, and the trip is logged ashelm.render.budget. No env knob was added.--set: an index past strvals' MaxIndex (65536), or a key longer than the parser holds, is no value, so the release is unknown. It never pads a list or splits a key.Reads by a built path (
pass_k8s.c,pass_terraform.c)Chart.yaml, kustomizevaluesFileor Terraformfile()is read only when it is a discovered file of the repository. So a symlink, or a path through one, is never followed out of it.Wrong images
setis Terraform's exact decimal (1.10→1.1), not%.17g.valuesFrom,chart.spec.valuesFiles,postRenderers;additionalValuesFiles, avaluesMergeother thanoverride, avaluesFilethat cannot be read;dynamic "set"..Chart.Nameset to its alias.if/with/rangemakes what its bodies assign UNKNOWN, and leaves the loop when abreakin them would.set/unset/mergepoison the map they change; a merge also poisons the maps within it.fromYaml,deepCopy, an emptydict, or a part dug out of one. Without that exception, 42 correct image edges on the infra corpora were lost; with it, none are.Marks and secrets
default_ofas JSON, so list paths likeimages[0].repositoryare kept. It is not written if its mark cannot be carried whole.cbm_is_secret_binding), are never kept.--setkeys are sized to their content, never cut.Proof on the 20 infra corpora
Tests
pipeline_semantic_manifest_repro, 10 probes ingrammar_probe_f.Verification
lint-no-suppress,lint-memory-core,scripts/security-audit.sh.Found in review round 1, not changed here
charts/<x>subchart is still a family member.default_of.{ref: image_default}) and D2 (the convention's host check): left for a decision.hy_nodeand the other parsers that walk the values recursively.hst_record/hy_mapping.seton.Valueschanges what later templates read in Helm; no render models that, and an include of a template file by path is the same case. The text atplrenders is not scanned.set_sensitivenames are not compared with the keys an image reads. (Done in review round 2, N9.)Review round 2: requested changes (N1-N12)
Every case below ends in an explicit unknown (no image), never in an OOM, a quadratic stall or a wrong image.
Bounds
helm_template.c): capture arrays grow by doubling, so n captures of a probe cost memory linear in n. Arena growth since the render began counts against a third budget, 256 MiB, beside steps and output. A trip logshelm.render.budgetwith its reason (steps,outputormemory). Release images are deduplicated by hash (pass_k8s.c), not by a list scan.dictper iteration keeps every later read O(1).Wrong images (
helm_template.c)set/mergein a body an UNKNOWN condition skips is judged by its target, whatever the target is. A call's result such asindex .Values "image"is evaluated in the frame: own, or a known map (now poisoned), is fine; anything else taints.setwhose target is an UNKNOWN call result (index .Values.apps .Release.Name) taints.break/continuespoils the rest of the loop body around it. Acontinueis now a continue, not a break.Releases and charts (
pass_k8s.c,pass_terraform.c)values.yamlis values{}, as in Helm, not a file that was not read.set_sensitivewhose name overlaps a key an image reads (adb.passwordone does not);postrenderblock;images,patches,patchesJson6902,patchesStrategicMerge,replacements,transformers,components.Tests
SKIP_PLATFORMon Windows), like every other symlink test. The Windows CI leg is the check.pipeline_semantic_manifest_repro, 3 probes ingrammar_probe_f(N9 a and b share one probe; both of its wrong edges show on 4cea54f).range 5000around an image linerange 20000range 20000of$c := dict+set $crange 40000Proof on the 20 infra corpora
Verification
lint-no-suppress,lint-memory-core,scripts/security-audit.sh.Found in review round 2, not changed here
hst_recordand in capture key matching.hy_noderecursion on deep YAML.setacross templates and the text atplrenders are still not modelled (as in round 1).Review round 3: the coarse rule (user decision)
The user chose a rule that is sound by construction over round 3's precise model: a change of maps the render does not run exactly leaves every later image of that render UNKNOWN. There are no exemptions. Its recall cost was measured before any push, and the user accepted it (table below).
The rule (
helm_template.c)Under UNKNOWN control, any of the following leaves the images after it UNKNOWN:
set/unset/merge, themust*forms included;tpl;UNKNOWN control means a skipped
if/with/rangebody, the rest of a loop an UNKNOWNbreak/continuemay leave, an include or template not run, or atplnot run.An executed
set/mergedoes the same unless its target is a known map that can be poisoned exactly. A merge also does it when the target is already poisoned (R3-7), is the.Valuesroot (R3-3), or holds a map that is not known.Round 3's own provenance and freshness scan are removed. R3-4, R3-5 and R3-6 came from them.
.Template.BasePath(approved by the user after the first measurement)<ChartFullPath>/templates. ChartFullPath is the top chart'sChart.yamlname, then/charts/<alias or name>for each subchart down.pass_k8s.ccomputes it where the alias is resolved..Template.Nameis not modelled (UNKNOWN).<chart dir>/templates/<rest>among the family's template files. That file is scanned with the same rule, memoized per render, and its output stays UNKNOWN. This is the checksum idiominclude (print $.Template.BasePath "/secret.yaml") . | sha256sum.Bounds
output, notmemory.Releases (
pass_terraform.c,pass_k8s.c)set_sensitiveleaves the release without an image of its own. Round 3's name-overlap check is removed.postrenderwritten as adynamicblock or as a provider-v3 attribute.Cost on the 20 infra corpora (this head against 5c5964e, whose graph equals d21949e's there; Apple M5 Pro, 2026-10-19)
mustMergeinto a values map the first merge poisoned. 8: jiralert, cloudwatch-exporter, postgres-exporter and yet-another-cloudwatch-exporter, each checksum-including a configmap/secret file that holdstplconfigmaps.yamlholdstplconfigmap.yamlholdstpltplmay change maps, because a tpl's text is not looked into.tplin it runs exactly, is a follow-up PR with its own measurement (user decision).Controls that flipped to UNKNOWN (part of the measured cost; their tests now say so)
helm_change_of_a_new_value_spoils_after(was..._spoils_nothing): changes offromYaml/deepCopy/empty-dictvalues. These kept their image in rounds 1-3, and it is the opentelemetry-ebpf-instrumentation pattern above.helm_skipped_set_after_redefinition, its control: asetof an outer newdictin a skipped body.probe_helm_release_terraform_sensitive_and_postrender: the release with adb.passwordset_sensitive(R3-8).Tests that fail on 5c5964e by their assertion
pipeline_semantic_manifest_repro:helm_capture_keys_cost_linear_memory(arena peak);helm_render_keeps_only_its_results;helm_render_budget_logs_its_reason;helm_merge_into_values_root;helm_skipped_set_beside_a_nested_scope;helm_unknown_break_spoils_later_elementsandhelm_skipped_range_sets_its_element;helm_deep_merge_reaches_a_held_values_map;helm_include_of_a_changing_template_file,helm_subchart_base_path_names_its_own_file,helm_include_of_a_missing_template_fileandhelm_template_file_including_itself;grammar_probe_f:probe_helm_release_terraform_sensitive_read_earlierand the flipped..._sensitive_and_postrender;probe_helm_release_terraform_postrender_forms;probe_helm_checksum_include_by_base_path.helm_checksum_include_keeps_the_imagefails as well, together with the BasePath tests (a)-(d) and the probe. 5c5964e did not taint a computed include, so it keeps that image.Corrections to the round-2 text (statements that were not true at 5c5964e)
pass_k8s.c, is still open (below).memory; it isoutputnow..Valuesroot path matched nothing. Now a merge into.Valuestaints.Verification
lint-no-suppress,lint-memory-core,scripts/security-audit.sh.Not changed here (a follow-up issue after the merge)
pass_k8s.c(k8s_helm_literal_use, the defaultssameloop).tpltrees outside the arena.breakin a skipped range'selse.first/last/joincopy the whole list.