Repository navigation
Imitate Format.AST proposal using bytecode decompiler - #130
Merged
Merged
Conversation
tests/_annos.py has 100 functions targeting Python 3.14 (PEP 649 deferred eval, PEP 695 type params): forward refs, unions, generics, callables, async types, Literal, Annotated, TypeGuard/TypeIs, ParamSpec, TypeVarTuple, Self, nested classes, protocols, TypedDict, and deeply nested types. tests/dump_annos.py enumerates all functions and prints three views of each annotation: the evaluated VALUE (annotationlib), the STRING form (from a temp copy with `from __future__ import annotations`), and the AST parse.
Stack-based interpreter that walks dis.Bytecode of __annotate__ methods and reconstructs equivalent ast.expr nodes. Handles both function annotations (BUILD_MAP N) and class annotations (BUILD_MAP 0 + STORE_SUBSCR). Passes 112/113 annotations from the test suite; the one miss is a lambda inside Annotated which needs a full Python decompiler. Library in typemap/type_eval/_decompile.py, test harness in tests/decompile_annos.py.
Handle the two control flow patterns CPython 3.14 generates for
conditional type annotations (e.g. `int if T else str`):
- Inline: TO_BOOL → POP_JUMP_IF_FALSE → true branch → JUMP_FORWARD →
false branch → join point. Produces ast.IfExp on the stack.
- Tail-position: each branch independently builds the annotation dict
and returns. Recursively decompile both branches and merge with
IfExp wrappers where values differ.
Handles nested if-expressions, complex conditions (e.g. IsAssignable),
and mixed conditional/non-conditional annotations in both function and
class contexts. Test suite now at 119 passed, 1 xfailed.
When CPython duplicates surrounding type context into both branches of a tail-position if-expression (e.g. list[int] vs list[str] for list[int if T else str]), the decompiler now pushes the IfExp down to the point of actual divergence instead of wrapping the entire expression. Handles Subscript, Tuple, BinOp, and Attribute with recursive structural comparison. Adds test cases for if-expressions nested inside list[], dict[], union (|), tuple[], and Callable[].
The tail-position branches can't be isolated from the pre-branch stack (e.g. list[int if T else str] puts list on the stack before branching, and both branches consume it). Go back to running both branches fully via _run, then structurally factoring IfExp into the deepest divergence point. Note: list[int] if T else list[str] generates identical bytecode to list[int if T else str] (CPython hoists list before the branch in both cases), so we always normalize to the inner form.
The bytecodes for list[int if T else str] and list[int] if T else list[str] ARE different (prefix context is on the stack before the branch in the former). However, postfix cases like (X if T else Y)|None produce identical bytecode to (X|None) if T else (Y|None), so we can't avoid factoring for those. Since we can't distinguish postfix-shared from deliberate-outer in the general case, we always factor — both forms are type-equivalent anyway.
X[A if C else B] ≡ X[A] if C else X[B], and similarly for postfix operators, so normalizing to the inner form is a correct transformation, not a lossy one.
Decompile unpacked comprehension types (*[x for x in Iter[T]]) used by PEP 827 type manipulation. Handles two bytecode patterns: - Pattern A (class/method annotations): separate <listcomp> code object called via MAKE_FUNCTION + CALL - Pattern B (module-level functions): comprehension inlined directly in the __annotate__ function with FOR_ITER loop New handlers: GET_ITER, LIST_EXTEND, LIST_APPEND, CALL_INTRINSIC_1. Comprehension filter clauses (if IsAssignable[...]) are also supported.
Extract _exec_stack_op() for shared stack opcodes (LOAD_*, BINARY_OP, BUILD_TUPLE/LIST, MAKE_FUNCTION, LIST_EXTEND/APPEND, CALL_INTRINSIC_1) and _handle_get_iter() for comprehension dispatch. Both _run, _run_expr, and _decompile_comp_body now delegate to these instead of reimplementing the same handlers. Net -108 lines.
- decompile_annos.py now auto-discovers all test .py files and runs the decompile-vs-string comparison on every annotation found - CLI accepts file paths: `python -m tests.decompile_annos path ...` - dump_annos.py gains collect_annotated() for generic annotation collection, load_stringified_copy() accepts arbitrary file paths - _is_compiler_annotate() detects the LOAD_FAST/LOAD_SMALL_INT 2 preamble to skip synthetic __annotate__ from TypedDict, dataclass, and NamedTuple metaclasses - Modules with `from __future__ import annotations` are skipped (PEP 563 bytecode is not what the decompiler targets)
When a listcomp inside __annotate__ captures a free variable from an enclosing class scope (e.g. T in __init_subclass__), the compiler emits SET_FUNCTION_ATTRIBUTE to attach closure cells. Pop the closure tuple and leave the function marker on the stack.
dis yields EXTENDED_ARG as a separate instruction (folding its value into the next instruction's arg), and jumps can target it. Drop it, mapping its offset to the following instruction.
The compiler folds e.g. -1 into a single constant; produce UnaryOp(USub, 1) as the source AST has it.
Add generic CALL (positional args only) and <genexpr> code objects, so calls like any(x for x in xs) decompile. The compiler inlines any()/all() over a generator expression behind a check that the name still refers to the builtin; follow the plain call path instead of the inlined loop. Factor the pc-changing ops shared by the interpreter loops (GET_ITER and the builtin guard) into _exec_op.
A loop variable used in a nested scope (like a generator expression in the comprehension body) is a cell, stored with STORE_DEREF.
LIST_EXTEND and LIST_APPEND mutated the target list node in place, but tail-position if-expression branches run from copies of the same stack, so items from both branches ended up in one shared list. Build a new node instead. Also merge starred items by merging their values, since an if-expression can't produce a starred item.
Bound both branches of an inline if-expression: the false branch of a nested one used to run on past its join point. And find the join from the jump before the false branch, since when the true branch itself ends in an if-expression, the optimizer threads that one's jumps straight to the outer join.
Sometimes we get back equivalent but non-identical ASTs. I didn't like any of the more elaborate tricks that the robot came up with for dealing with them so...
Merge tail-position if-expression branches generically, which fixes factoring inside calls, comparisons, not, and lists. Unify the comprehension paths, tighten some checks, refresh docs, and rename BindingMetadata to BindingNamespace.
BUILD_MAP was always treated as building the annotations dict, so a dict display inside an annotation was silently decompiled into bogus annotations. Build an ast.Dict instead, and only read annotations from the dict an __annotate__ function returns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement a bytecode decompiler for annotate functions that can
handle the annotations that PEP 827 uses (except GenericCallable,
which I am going to get rid of).
This is more slop than I usually aim for.
Dealing with all of the things traditionally stored in annotations was
pretty easy. The tricky parts were all in conditionals and loops.
Conditionals fork and run both branches until either they join back up
or both exit. We then do a bunch of annoying and somewhat ad-hoc stuff
to try to produce good ASTs for them:
list[t if b else f]fromlist[t] if b else list[f], since those will sometimes compileto the same bytecode.
andandor, etc. I had a version that tried much harder toalways produce the original ast, by also looking at source spans,
but decided it wasn't worth the trouble.