Repository navigation
Conversation
ashb
added this pull request to stack #74339
October 6, 2026 15:06
kaxil
reviewed
Oct 6, 2026
ashb
force-pushed
the
task-loops-stack-6
branch
from
October 7, 2026 13:57
d25a7c8 to
2e69b13
Compare
ashb
removed this pull request from stack #74339
October 7, 2026 15:20
ashb
force-pushed
the
task-loops-stack-6
branch
from
October 7, 2026 15:22
2e69b13 to
7be9b12
Compare
ashb
added this pull request to stack #74410
October 7, 2026 15:22
ashb
force-pushed
the
task-loops-stack-6
branch
2 times, most recently
from
October 7, 2026 20:57
8a05c82 to
6bb15d6
Compare
ashb
force-pushed
the
task-loops-stack-6
branch
from
October 8, 2026 13:48
6bb15d6 to
5c17d4d
Compare
ashb
force-pushed
the
task-loops-stack-6
branch
2 times, most recently
from
October 8, 2026 16:06
0e415ae to
c3b6ed7
Compare
kaxil
approved these changes
Oct 8, 2026
ashb
force-pushed
the
task-loops-stack-6
branch
from
October 8, 2026 20:45
c3b6ed7 to
8d6022b
Compare
ashb
requested review from
XD-DENG,
amoghrajesh,
bbovenzi,
bugraoz93,
choo121600,
dheerajturaga,
guan404ming,
henry3260,
hussein-awala,
jason810496,
o-nikolas,
pierrejeambrun,
potiuk,
rawwar,
ryanahamilton,
shubhamraj-git and
vatsrahul1001
as code owners
October 9, 2026 12:39
ashb
force-pushed
the
task-loops-stack-6
branch
from
October 9, 2026 14:03
aedf184 to
70a0e6f
Compare
ashb
force-pushed
the
task-loops-stack-6
branch
2 times, most recently
from
October 9, 2026 20:34
1f39e86 to
eda70b1
Compare
ashb
force-pushed
the
task-loops-stack-6
branch
from
October 9, 2026 20:46
eda70b1 to
ec16149
Compare
ashb
force-pushed
the
task-loops-stack-6
branch
from
October 9, 2026 22:18
ec16149 to
0f448fb
Compare
Audit log rows, asset events and asset-state writers identified their task by (dag_id, task_id, run_id, map_index). Once a loop pass or a clear can reuse those coordinates for another execution, a row read back by coordinates points at the wrong attempt. Retired attempts stay in task_instance under their own UUID, so the UUID is the strongest anchor: a row resolves to the attempt that actually acted, and keeps doing so after that attempt is retired. Rows written before the upgrade only have coordinates. We leave them without an attributed attempt instead of matching them to a guess, because a wrong attribution is worse than none. An attributed row's public map index comes from the attempt's pinned definition, as it does everywhere else, and is NULL once the attempt row has been purged because nothing is left to derive it from. The asset-state writer also records its region id, region index and try number so it stays describable after cleanup removes the attempt. The stuck-in-queued accounting counts log rows by attempt UUID and falls back to coordinates only for sentinel-region rows, which are the rows that can still be unambiguous by coordinates. OpenLineage uses the attempt UUID as the run id of regional tasks so the listener events, the lineage macros and downstream asset dependencies agree on which run they describe. Task notes need no new storage: a note hangs off the attempt UUID and a retired attempt keeps its row, so the retain-notes migration from the earlier design is not carried over.
ashb
force-pushed
the
task-loops-stack-6
branch
from
October 10, 2026 07:05
0f448fb to
094f4b6
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Audit log rows, asset events and asset-state writers identified their task by
(dag_id, task_id, run_id, map_index). Once a loop pass or a clear can reuse
those coordinates for another execution, a row read back by coordinates points
at the wrong attempt. Retired attempts stay in task_instance under their own
UUID, so the UUID is the strongest anchor: a row resolves to the attempt that
actually acted, and keeps doing so after that attempt is retired.
Rows written before the upgrade only have coordinates. We leave them without
an attributed attempt instead of matching them to a guess, because a wrong
attribution is worse than none.
An attributed row's public map index comes from the try's pinned definition,
as it does everywhere else, and is NULL once the attempt row has been purged
because nothing is left to derive it from. The asset-state writer also records
its region id, region index and try number so it stays describable after cleanup
removes the attempt. The stuck-in-queued accounting counts log rows by attempt
UUID and falls back to coordinates only for sentinel-region rows, which are the
rows that can still be unambiguous by coordinates.
OpenLineage uses the attempt UUID as the run id of regional tasks so the
listener events, the lineage macros and downstream asset dependencies agree on
which run they describe.
Task notes need no new storage: a note hangs off the attempt UUID and a retired
attempt keeps its row, so the retain-notes migration from the earlier design is
not carried over.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.