Enable ACL confidential VM boot and harden Azure time synchronization by mayankfz · Pull Request #61 · microsoft/azure-container-linux

mayankfz · 2026-08-22T19:20:21Z

Summary

Enable ACL confidential VM boot and harden Azure time synchronization
doc - https://microsoftapc-my.sharepoint.com/:w:/r/personal/mayansingh_microsoft_com/_layouts/15/Doc.aspx?sourcedoc=%7Bebae3a07-f597-4c65-96f0-143459146fb9%7D&action=default

Summary
Adds Azure Confidential VM support and hardens Azure time synchronization.

Adds TPM2 dependencies required for encrypted ROOT auto-unlock.
Preserves ignition-disks -> decrypt-root -> initrd-root-device ordering.
Uses Hyper-V PTP when available without delaying or breaking non-PTP VMs.
Keeps time.windows.com as the documented public fallback.
Makes Chrony image customization fail clearly when expected inputs drift.

Change Log

Enable ACL confidential VM boot and harden Azure time synchronization

Type of Change

Does this affect the image build?

Yes
No

Associated Issues

https://dev.azure.com/mariner-org/ACL/_workitems/edit/23679

Test Methodology

Test details:

Normal Azure VM boot and reboot passed.
Chrony passed with and without ptp_hyperv.
Encrypted ROOT first boot passed on AMD SEV-SNP.
Azure vTPM enrollment using PCR 7 passed.
Two unattended reboots passed after removing all passphrase slots.
Intel TDX validation will run through the AKS image pipeline.

Merge Checklist

All applicable boxes should be checked before merging

Image builds successfully with this change (or image build is not affected)
Any updated packages/SPECs build successfully
Relevant kola tests pass
All package sources are available
Source files have up-to-date hashes/manifests
Documentation has been updated to match any changes
Ready to merge

Jiri Appl (jiria) · 2026-09-01T14:28:17Z

Is this all we were missing to unlock encrypted rootfs on boot?

mayankfz · 2026-09-01T14:45:00Z

Is this all we were missing to unlock encrypted rootfs on boot?

These were the remaining blockers in our validation, the TPM2 libraries needed for auto-unlock and the decrypt-root ordering fix. The LUKS/dracut support was already added in #33. With these changes, encrypted ROOT now unlocks and boots successfully in our CVM test.

Jiri Appl (jiria) · 2026-09-01T15:02:19Z

While this is still in draft — could you fill in the description before marking it ready? Right now it's the unmodified template: no change-type ticked, empty "Test details:", and no linked issue.

Given this touches initramfs root decryption and time sync on every Azure image, the things I'd most want recorded are:

The actual Found ordering cycle journal lines that motivated the bootengine patch. That's the whole justification for the ordering change, and without it the next person to touch these units is flying blind.
Boot results on both an encrypted CVM (the fix path) and a non-CVM (the regression path).
chronyc sources on a PTP-capable machine and on one without /dev/ptp_hyperv.

Also worth revisiting the title — "Add ACL CVM support" undersells it a bit, since four of the seven files change chrony behaviour on every Azure image rather than just CVMs.

Copilot

🟡 Changes recommended

The initramfs TPM2 library inclusion is currently /usr/lib64-only, which can omit required libs on non-lib64 layouts and break LUKS root auto-unlock.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR enables Azure Confidential VM boot support (TPM-backed LUKS root auto-unlock) and hardens Azure time synchronization by making Hyper-V PTP optional/non-blocking while retaining a public NTP fallback.

Changes:

Add a chrony helper (chrony-azure-ptp) and systemd drop-in to generate PTP chrony config at service start when /dev/ptp_hyperv exists.
Add TPM2 (tss2) shared libraries to the initramfs explicit install list to support runtime dlopen() by systemd-cryptsetup.
Patch bootengine’s decrypt-root.service ordering to run before initrd-root-device.target.

File summaries

File	Description
sdk_container/src/third_party/coreos-overlay/coreos-base/oem-azure/files/manglefs_rpm.sh	Adjusts chronyd unit customization, removes unsupported ExecStopPost helper, and installs the Azure PTP helper + drop-in.
sdk_container/src/third_party/coreos-overlay/coreos-base/oem-azure/files/chrony.conf	Switches to include Azure runtime-generated chrony config and reintroduces `time.windows.com` fallback.
sdk_container/src/third_party/coreos-overlay/coreos-base/oem-azure/files/chrony-hyperv.conf	Updates chronyd drop-in to avoid waiting for Hyper-V PTP while still ordering after it when present, and runs PTP config generator.
sdk_container/src/third_party/coreos-overlay/coreos-base/oem-azure/files/chrony-azure-ptp	New helper script that writes/removes the PTP refclock config under `/run/chrony-azure/conf.d`.
build_library/rpm/additional_files/99-acl.conf	Adds explicit TPM2 shared library inclusion for initramfs root auto-unlock support.
acl/SPECS/bootengine/bootengine.spec	Bumps release and applies new decrypt-root ordering patch.
acl/SPECS/bootengine/0003-decrypt-root-fix-firstboot-ordering.patch	Ensures decrypt-root runs before `initrd-root-device.target` to preserve firstboot ordering.

Review details

Files reviewed: 7/7 changed files
Comments generated: 3
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🔵 Needs a closer look

It changes early-boot/initramfs composition and systemd ordering for encrypted-root bring-up, which is high impact and best gated by final human review.

Review details

Files reviewed: 8/8 changed files
Comments generated: 0 new
Review effort level: Lite

Use the signed ACL production Azure Compute Gallery images for amd64, arm64, and CVM VHD builds. Pin amd64 to 20260923.1209152.0 and arm64 to 20260923.1209110.0. Route ACL CVM through the amd64 ACG source instead of Marketplace coordinates. Preserve empty Marketplace SKU arguments in the content tests so ACL retains its cron-permission exemption. Prefix the optional IMG_SKU and FEATURE_FLAGS values at the Run Command boundary, then remove those prefixes in the VM script. This prevents empty arguments from shifting the commit SHA and repository URL while preserving exact-commit checkout and legacy unprefixed callers. Add transport round-trip regressions for empty values, ACL/FIPS/CVM and non-ACL flags, literal prefixes, custom repository URLs, and the legacy repository default. Temporarily skip testChrony only for ACL images with the exact cvm feature flag, before any service checks or clock mutation. Log the waiver explicitly and retain the package endpoint test and all other image validation. Remove this exception once the pinned ACL base image includes microsoft/azure-container-linux#61. Add 19 Chrony regression cases covering the real call site, combined flags, non-CVM ACL, other OSes, and preserved service/time-correction failures. The four waiver cases fail before the change; all 56 focused examples pass afterward. Bash lint and syntax checks pass with no new POSIX diagnostics. Signed-off-by: Aadhar Agarwal <aadagarwal@microsoft.com>

Use the signed ACL production Azure Compute Gallery images for amd64, arm64, and CVM VHD builds. Pin amd64 to 20260923.1209152.2 and arm64 to 20260923.1209110.1. Route ACL CVM through the amd64 ACG source instead of Marketplace coordinates. Preserve empty Marketplace SKU arguments in the content tests so ACL retains its cron-permission exemption. Prefix the optional IMG_SKU and FEATURE_FLAGS values at the Run Command boundary, then remove those prefixes in the VM script. This prevents empty arguments from shifting the commit SHA and repository URL while preserving exact-commit checkout and legacy unprefixed callers. Add transport round-trip regressions for empty values, ACL/FIPS/CVM and non-ACL flags, literal prefixes, custom repository URLs, and the legacy repository default. Temporarily skip testChrony only for ACL images with the exact cvm feature flag, before any service checks or clock mutation. Log the waiver explicitly and retain the package endpoint test and all other image validation. Remove this exception once the pinned ACL base image includes microsoft/azure-container-linux#61. Add 19 Chrony regression cases covering the real call site, combined flags, non-CVM ACL, other OSes, and preserved service/time-correction failures. The four waiver cases fail before the change; all 56 focused examples pass afterward. Bash lint and syntax checks pass with no new POSIX diagnostics. Use ACL publish run 1214376, which reused the exact bundle tags and digests from run 1209759 and published to all destinations with West Europe included in gallery replication. Signed-off-by: Aadhar Agarwal <aadagarwal@microsoft.com>

Use signed production ACG images replicated to West Europe: amd64 20260923.1209152.2, arm64 20260923.1209110.1. Preserve empty test arguments and temporarily skip ACL CVM Chrony until microsoft/azure-container-linux#61 is included. Signed-off-by: Aadhar Agarwal <aadagarwal@microsoft.com>

…t merge) Experiment only, do not merge. Builds the ACL CVM VHD (buildaclcvmgen2) on the microsoft/azure-container-linux#61 PR validation image 035db282-f1c8-4ce7-b78f-2a7265d5398c-ACLDEVEL/acldevel-gh-amd64/0.20261007.1219462 to check whether the chrony fix clears the known testChrony failures. The PR image's UKI is dev-signed, so the CVM test and scan VMs boot with Secure Boot off. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

The changes span confidential-VM boot, initramfs ordering, TPM2 dependencies, and time synchronization; final human validation is warranted.

0 open findings

1 resolved since last review

Handle empty TPM2 grep results to preserve cleanup and diagnostics

🧠 Review effort: Lite

mayankfz · 2026-10-07T18:02:21Z

Could we make the initramfs TPM-library check fail if it extracts no required libraries from 99-acl.conf? The current six /usr/lib64/libtss2- paths match, but if a later config or package change moves or renames them, the while loop would check nothing and leave missing_tss empty. The build caller uses set -e without pipefail, so this would report success rather than detect that the verification list has gone empty. Checking that the extracted list is nonempty before inspecting the archive would keep this drift guard fail-closed.

Thanks, have updated now to extracts the list and checks it's non-empty before touching the archive, failing through the existing cleanup path.

Jiri Appl (jiria) · 2026-10-07T18:28:42Z

The new time.windows.com fallback omits maxdistance 16.0, which Azure Linux 3.0's Chrony package adds to tolerate that server's delay. Chrony's default limit is 3 seconds, so a non-PTP CVM could reject the fallback if its root distance exceeds that limit and no other source is usable. Could we either retain the packaged setting or check chronyc sources -v and chronyc tracking on a non-PTP CVM to confirm the tighter limit is safe? maxdistance applies globally, so raising it would also affect acceptance of other sources, including DHCP.

Jiri Appl (jiria) · 2026-10-07T18:28:46Z

The new initramfs check searches lsinitrd output for each TSS SONAME as a substring. For example, an entry named libtss2-esys.so.0.0.0 would satisfy a search for libtss2-esys.so.0 even if the SONAME symlink were missing, though that exact name is needed at runtime. Could the assertion match complete archive entry paths and verify the symlink targets are present? The current install list requests these links, so this is a gap in the build-time check, not evidence that this image is missing them.

Copilot

🟡 Changes recommended

The encrypted-root dependency ordering has a critical unresolved issue, and the changed OEM package needs a revision bump.

1 open finding

Ensure root target waits for decrypt-root.service · New

🧠 Review effort: Lite

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Copilot

🔵 Needs a closer look

The changes require final human review because they are too complex or risky for automated approval.

0 open findings

1 resolved since last review

Ensure root target waits for decrypt-root.service

🧠 Review effort: Lite

Jiri Appl (jiria) · 2026-10-09T04:07:56Z

Could we coordinate the downstream image customization update before it consumes this change? Both chrony.conf and chrony-hyperv.conf change here, but the downstream customization currently checks their stock SHA-256 values against the previous versions and stops on a mismatch. Once it consumes the merged branch, that guard will prevent the affected image variants from completing customization.

Please re-audit the downstream Chrony overrides against the new PTP source generation, then update the stock-file pins in coordination with this PR rather than refreshing the hashes alone.

mayankfz · 2026-10-09T09:13:15Z

Considering that this is just an append, you could use a drop-in conf file. That way, you don't need to maintain a diff file.

Now we are retaining the TPM2 dracut module, so I've dropped both the explicit install_items and the assertion entirely. Nothing left to check.

mayankfz · 2026-10-09T09:17:37Z

The new time.windows.com fallback omits maxdistance 16.0, which Azure Linux 3.0's Chrony package adds to tolerate that server's delay. Chrony's default limit is 3 seconds, so a non-PTP CVM could reject the fallback if its root distance exceeds that limit and no other source is usable. Could we either retain the packaged setting or check chronyc sources -v and chronyc tracking on a non-PTP CVM to confirm the tighter limit is safe? maxdistance applies globally, so raising it would also affect acceptance of other sources, including DHCP.

Measured it on a VM, time.windows.com reports root delay 61us / dispersion 10.2ms, so ~10ms root distance against chrony's 3s default. Setting maxdistance 16.0 explicitly changed nothing, so I've left it off rather than widen acceptance for every source.

mayankfz · 2026-10-09T09:19:11Z

The new initramfs check searches lsinitrd output for each TSS SONAME as a substring. For example, an entry named libtss2-esys.so.0.0.0 would satisfy a search for libtss2-esys.so.0 even if the SONAME symlink were missing, though that exact name is needed at runtime. Could the assertion match complete archive entry paths and verify the symlink targets are present? The current install list requests these links, so this is a gap in the build-time check, not evidence that this image is missing them.

Now we are retaining the TPM2 dracut module, so I've dropped both the explicit install_items and the assertion entirely. Nothing left to check.

mayankfz · 2026-10-09T10:40:11Z

Could we coordinate the downstream image customization update before it consumes this change? Both chrony.conf and chrony-hyperv.conf change here, but the downstream customization currently checks their stock SHA-256 values against the previous versions and stops on a mismatch. Once it consumes the merged branch, that guard will prevent the affected image variants from completing customization.

Please re-audit the downstream Chrony overrides against the new PTP source generation, then update the stock-file pins in coordination with this PR rather than refreshing the hashes alone.

Rather than just refreshing the hashes, acl-t's refclock now gets trust: without it a disagreement between the host clock and the network source marks both falsetickers and chronyd stops syncing entirely (reproduced on a VM). Pins updated to match. Needs to merge alongside the current PR since they track those exact files.
PR -> https://dev.azure.com/mariner-org/ACL/_git/acl-pipelines/pullrequest/29682

pipeline ->
acldevel -> https://dev.azure.com/mariner-org/ACL/_build/results?buildId=1220834&view=results
aks-image-build -> https://dev.azure.com/mariner-org/mariner/_build/results?buildId=1220900&view=results