Repository navigation
Conversation
Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>
Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>
Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>
Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Add type annotation stubs for array, table, tensor, builder, memory, device, config, and types modules. Includes type-ignore annotations in related tests.
Drop the TypeVar/Generic parameters from the DataType subclasses, Field, Array, ChunkedArray and the tabular classes. Where a property can only return one of a few types (dictionary index type, run end type, time units) it now returns a union or Literal instead. Most of these parameters weren't buying much yet: scalar.pyi and lib.pyi are still placeholders, so e.g. Array[Scalar[...]] collapsed to Any anyway, and several annotations were being flagged by mypy and pyright (TypeVars used only once, type arguments on subclasses that weren't actually generic). Plain classes are a lot easier to read and review, and we can bring parameters back later where they clearly pay off. This also removes the stub-only _BasicDataType and _HasPrecisionScale helpers, which have no runtime counterpart, plus the IntoArray and SingleOrList aliases.
Add pyarrow-stubs/tests with pandas-stubs style checks:
check(assert_type(lib.list_(lib.int32()), lib.ListType), lib.ListType)
assert_type is verified by mypy, pyright and ty, which already include
pyarrow-stubs/ in their runs, and check() verifies the runtime class when
the tests run under pytest, so the stubs can't silently drift from what
pyarrow returns. Usage that should be rejected goes under
`if TYPE_CHECKING_INVALID_USAGE:` with a `# type: ignore[...]`; unused
ignores are errors in these tests, so those fail if the usage starts
type checking.
To have something to test, lib.pyi now re-exports the stub modules that
are annotated so far, still falling back to Any for everything else.
The CI type annotation script runs the tests with pytest after the type
checkers.
|
|
table.pyi imports RecordBatchReader from ._ipc, which had no stub, so pyright and ty failed to resolve the import. Add a placeholder in the same style as io.pyi and scalar.pyi until _ipc gets fully annotated.
compute and interchange will be annotated in later PRs, so treat imports of them as Any instead of type checking their untyped sources (mypy) or failing to resolve them (ty). Recent ty versions also no longer honor `# type: ignore[<mypy code>]` comments, so disable the rules those comments suppress: method overrides in array.pyi, and invalid arguments in the tests, where mypy and pyright still check that the invalid usage is rejected.
- TypesEnum: list all members, with the runtime values. - is_nested() also matches run-end encoded types. - Arrays and chunked arrays have no as_py() method. - ChunkedArray.unique() returns an Array. - MapArray is a ListArray, so it has offsets and values. - value_lengths() of large list (view) arrays returns an Int64Array. - Array.from_pandas() arguments can be passed positionally. Add annotation tests for these.
Converting timedelta.max to a millisecond or second duration truncates the sub-unit part, so it doesn't round trip. The comparisons used to be missing their assert, which hid this.
An empty placeholder class rejects valid usage such as table.to_reader().read_all(), as it has no methods.
Decimal types subclass FixedSizeBinaryType, but is_fixed_size_binary() returns False for them. With TypeIs, type checkers would wrongly narrow decimal types away when the check fails.
Run ruff, mypy, pyright and ty on the stubs with pinned versions: pre-commit run --hook-stage manual python-stubs --all-files The hooks only run on request, so they don't affect the existing hooks or CI jobs.
|
Thanks for continuing to push this forward Rok! I think this PR is a great next step and is quite clear to review. I have checked out the PR locally and started with running type checkers ( The reason for these warnings and errors in I propose to push the changes directly if we agree the additions makes sense. Meanwhile I will continue with running the |
|
I have run I have also run mypy's stubtest with Besides the reports mentioned above which will be fixed in subsequent PRs, stubtest did found one bug in pyarrow: Note missing "L" in "Decimal". I will work on the fix for this now and would suggest to add it as part of this PR. Can commit directly, if that is ok. |
|
I've run Running on editable dev build 27, with pyright==1.1.414 in my conda env, I also hit errors |
That makes sense and please go for it. I think we should aim for
Huh. I've never noticed these were missing. Given we're not really using them as types I'd skip them if we can.
Huh. Let's do the fix as separate PR, but also include a fix here? (And we'll just rebase over the actual fix on main)
I'm not sure we want (can) bring in ruff as a CI dependency. However we should autoformat before we merge, so we don't do it later and change history for no good reason. So I propose one of you runs the ruff formatter and commits the change into this branch. |
I think
Sounds good. Will open an issue and a PR and add a fix here also. |
Yeah, they are internal and exposed through |
Ok, I'll do that reformatting in a bit. |
Rationale for this change
This is the next step in adding type annotations to PyArrow (#32609). It merges #49103 (internal type system stubs) and #49193 (core data structures) because the two depend on each other and can't be type checked separately.
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?
Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: