Skip to content

Add Table::describe #3239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
texodus merged 1 commit into from
Sep 24, 2026
+7,511 −2,257
Merged

Add Table::describe #3239

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
206 changes: 168 additions & 38 deletions DEVELOPMENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,20 +12,48 @@ might not work the way you're used to!

Perspective is organized as a
[monorepo](https://github.com/babel/babel/blob/master/doc/design/monorepo.md),
and uses [lerna](https://lernajs.io/) to manage dependencies.

This guide provides instructions for both the JavaScript and Python libraries.
To switch your development toolchain between the two, use `pnpm run setup`. Once
the setup script has been run, common commands like `pnpm run build` and
`pnpm run test` automatically call the correct build and test tools.
and uses [pnpm workspaces](https://pnpm.io/workspaces) to manage dependencies.
All commands in this guide are run from the repository root.

| Path | Contents |
| ------------------------------------------------------------- | -------------------------------------------------------------------- |
| `rust/perspective-server` | The C++ engine, compiled natively and to WebAssembly via Emscripten |
| `rust/perspective-client` | Rust client, `perspective.proto` and the Virtual Server framework |
| `rust/perspective-js` | `@perspective-dev/client`, the JavaScript/WebAssembly bindings |
| `rust/perspective-python` | `perspective-python`, the [PyO3](https://pyo3.rs) bindings |
| `rust/perspective-viewer` | `@perspective-dev/viewer`, the `<perspective-viewer>` Custom Element |
| `rust/perspective` | The `perspective` Rust crate |
| `packages/viewer-datagrid`, `packages/viewer-charts` | Viewer plugins |
| `packages/react`, `packages/jupyterlab`, `packages/anywidget` | Framework and notebook integrations |
| `tools/scripts`, `tools/test`, `tools/bench` | Build scripts, the shared test harness and the benchmark suite |
| `examples`, `docs` | Example projects and the documentation site |

This guide provides instructions for the JavaScript, Python and Rust libraries.
To choose which packages your development toolchain builds and tests, use
`pnpm run setup`. Once the setup script has been run, common commands like
`pnpm run build` and `pnpm run test` automatically call the correct build and
test tools for the selected packages.

### System Dependencies

`Perspective.js` and `perspective-python` **require** the following system
dependencies to be installed:

- [Node.js](https://nodejs.org/) (version 22 is what CI uses)
- [pnpm](https://pnpm.io/)
- [Rust](https://rustup.rs/) via `rustup`. The pinned nightly toolchain and
WebAssembly targets in `rust-toolchain.toml` are installed automatically.
- [CMake](https://cmake.org/) (version 3.29.5 or higher)
- [pnpm](https://pnpm.io/).
- A C++17 compiler for native builds (`perspective-python` and the Rust crate).
LLVM 17 is the pinned version, which `pnpm run install_llvm` will download to
`.llvm/`.

Running `pnpm install` additionally downloads the pinned versions of
[Emscripten](https://emscripten.org/) and
[Binaryen](https://github.com/WebAssembly/binaryen) specified in `package.json`,
and the Chromium build used by [Playwright](https://playwright.dev/). Boost and
the other C++ dependencies are downloaded by CMake at build time, and do not
need to be installed.

**_This list may be non-exhaustive depending on your OS/environment; please open
a thread in
Expand All @@ -51,18 +79,44 @@ a `.perspectiverc` via a short survey. This can be later re-configured via
pnpm run setup
```

`.perspectiverc` is a plain `KEY=value` file, and any of its values can be
overridden per-command from the environment. `PACKAGE` is a comma-separated list
of the package names shown by `pnpm run setup`, e.g. to build just the engine
and JavaScript client:

```bash
PACKAGE=server,client pnpm run build
```

Note that `PACKAGE` is a filter, not a dependency graph. Packages which are not
selected are not rebuilt, so e.g. a change to the C++ engine in
`rust/perspective-server` will not be reflected in `@perspective-dev/client`
unless `server` is also selected.

Other useful options:

| Variable | Effect |
| --------------------- | ---------------------------------------------------------- |
| `PSP_DEBUG=1` | Debug build |
| `PSP_BUILD_VERBOSE=1` | Verbose C++ build output |
| `PSP_NUM_CPUS=<n>` | Limit C++ build parallelism |
| `PSP_WASM64=1` | Also build the `wasm64` (Memory64) engine |
| `PSP_BUILD_WHEEL=1` | Build a `perspective-python` wheel to `rust/target/wheels` |
| `PSP_DOCKER=1` | Build inside the Docker build environment |

If everything is successful, you should be able to run any of the `examples/`
packages, e.g. `examples/blocks` like so:
packages, e.g. `examples/esbuild-example` like so:

```bash
pnpm run start blocks
pnpm run start esbuild-example
```

To remove build artifacts, use `pnpm run clean`.

## `Perspective.js`

To build the JavaScript library, which includes WebAssembly compilation,
[Emscripten](https://github.com/kripken/emscripten) and its prerequisites are
required.
[Emscripten](https://emscripten.org/) and its prerequisites are required.

`Perspective.js` specifies its Emscripten version dependency in `package.json`,
and the correct version of Emscripten will be installed with other JS
Expand All @@ -84,12 +138,15 @@ Deviating from this specific version of Emscripten specified in the project's
`package.json` can introduce various errors that are extremely difficult to
debug.

To install a specific version of Emscripten (e.g. `2.0.6`):
To install a specific version of Emscripten (e.g. `4.0.9`):

```bash
./emsdk install 2.0.6
./emsdk install 4.0.9
```

Set `PSP_SKIP_EMSDK_INSTALL=1` to prevent `pnpm install` from downloading the
bundled version.

---

## `perspective-python`
Expand All @@ -102,27 +159,43 @@ of python, e.g.
pip install -r rust/perspective-python/requirements.txt
```

`pnpm run build` will then compile the extension and install it into your active
Python environment in development mode via
[`maturin develop`](https://www.maturin.rs/). It is strongly recommended to do
this within a virtual environment.

`perspective-python` supports Python 3.11 and upwards.

To build for [Pyodide](https://pyodide.org/), select
`perspective-python (pyodide)` in `pnpm run setup` and install the pinned
Pyodide distribution with `pnpm run install_pyodide`.

### `perspective-jupyterlab`

To install the Jupyterlab/Jupyter Notebook plugins from your local working
directory, simply install `python/perspective` with `pip` as you might normally
do.
The JupyterLab extension is built by the `jupyterlab` package, which copies the
resulting labextension into the `perspective-python` package's data directory.
To install it from your local working directory, build both packages as a wheel
and install the wheel with `pip`:

```bash
# builds labextension to the perspective-python python package root directory
PACKAGE=perspective-jupyterlab pnpm run build
# editable install of the python package
pnpm -F @perspective-dev/python develop:maturin
# set up symlink of our labextension to jupyter share directory
# this directory's path is in the output of `jupyter labextension list`
pnpm -F @perspective-dev/python develop:labextension
# builds the labextension, then a wheel which bundles it
PACKAGE=jupyterlab,python PSP_BUILD_WHEEL=1 pnpm run build
pip install --force-reinstall rust/target/wheels/perspective_python-*.whl
```

Afterwards, you should see it listed as a "local extension" when you run
`jupyter labextension list` and as a normal extension when you run
`jupyter nbextension list`.
Afterwards, you should see `@perspective-dev/jupyterlab` listed when you run
`jupyter labextension list`.

## `perspective` (Rust)

To build the Rust crate, select `perspective (rust)` in `pnpm run setup`.

The root `.cargo/config.toml` shares one `target-dir` (`rust/target`) between
the native and WebAssembly builds. When invoking `cargo` directly, always pass
an explicit `--target` (e.g. `--target=wasm32-unknown-unknown` for
`perspective-viewer` and `perspective-js`), otherwise `cargo` will fingerprint
shared host artifacts differently than `pnpm run build` does, and each will
invalidate the other's cache.

---

Expand All @@ -137,7 +210,7 @@ brew install cmake llvm@17
brew link llvm@17 # optional, see below
```

On M1 (Apple Silicon) systems, make sure your brew-installed dependencies are in
On Apple Silicon systems, make sure your brew-installed dependencies are in
`/opt/homebrew` (the default location), and that `/opt/homebrew/bin` is on the
`PATH`.

Expand Down Expand Up @@ -168,39 +241,94 @@ prerequisite tools.

### Ubuntu/Debian

On Ubuntu, CMake will mistakenly resolve the system headers in `/usr/include`
rather than the emscripten supplied versions. You can resolve this by moving
`boost` dependencies to somewhere other than `/usr/include` - into Perspective's
own `src` dir (as per
[here](http://vclf.blogspot.com/2014/08/emscripten-linking-to-boost-libraries.html)).
Install system dependencies through `apt`:

```bash
apt-get install libboost-all-dev
cp -r /usr/include/boost ./packages/perspective/src/include/
apt-get install build-essential cmake
```

Boost is downloaded by CMake at build time; a system `libboost` is not required.

---

## Test

You can run the test suite simply with the standard NPM command, which will both
build the test suite for every package and run them.
You can run the test suite for the packages selected in `.perspectiverc` with
the standard NPM command.

```bash
pnpm run test
```

The test suite runs against the artifacts of the last `pnpm run build`, and does
not rebuild them; remember to re-run the build after making a change.

### JavaScript

The JavaScript test suite is composed of two sections: a Node.js test, which
asserts behavior of the `@perspective-dev/client` library, and a suite of
[Playwright](https://playwright.dev/) tests, which assert the behavior of the
rest of the UI facing packages.

`PACKAGE` selects which packages' suites run, and extra arguments are forwarded
to Playwright, so to run a single spec or test:

```bash
PACKAGE=viewer-datagrid pnpm run test column_style.spec
PACKAGE=client pnpm run test -g "to_arrow"
```

Each package is a Playwright project named `<package>-desktop-chrome` (or
`<package>-node` for the Node.js suites), which can be passed as `--project` to
narrow a run when several packages are selected.

Set `PSP_HEADED=1` to watch the browser tests run. The JupyterLab integration
tests are run with `PACKAGE=jupyterlab pnpm run test --jupyter`.

Many UI tests compare against screenshot and DOM snapshots, which live in
`tools/test/dist/snapshots` and are not checked in to this repository. CI
fetches them from a separate snapshots repository. To regenerate
snapshots locally after an intentional rendering change, or to generate them
for the first time from a known-passing (in CI) build you've checked out:

```bash
pnpm run test --update-snapshots
```

Locally regenerated snapshots only affect your machine; a pull request which
changes rendering also needs its snapshots veriied and published to the
snapshots repository by a maintainer.

### Python

With `python` selected, `pnpm run test` runs the `pytest` suite; extra arguments
are forwarded to `pytest`.

### Rust

With `rust` selected, `pnpm run test` runs `cargo test` for the `perspective`
and `perspective-client` crates.

## Lint

```bash
pnpm run lint
pnpm run fix
```

`lint` checks license headers, `eslint`, `prettier`, `clippy` and `rustfmt` (and
`ruff` when `perspective-python` is selected), and is run as a pre-push hook.
`fix` applies the automatic fixes.

## Docs

```bash
pnpm run docs
```

Documentation sources live in `docs/md`. This runs `cargo doc`, then the `docs`
script of each selected package.

### Troubleshooting installation from source

If you are installing from a source distribution (sdist), make sure you have the
Expand All @@ -215,15 +343,17 @@ pip install -vv perspective-python
The most common culprits are:

- CMake version is too old
- Boost headers are missing or too old
- No C++17 compiler is available

---

## Benchmark

You can generate benchmarks specific to your machine's OS and CPU architecture
with Perspective's benchmark suite, which will host a live dashboard at
http://localhost:8080 as well as output a result `benchmark.arrow` file.
http://localhost:8080 as well as output a result `.arrow` file to
`tools/bench/dist`. The suite which runs is chosen by the selected packages:
`viewer-charts`, `client` or `python`.

```bash
pnpm run bench
Expand Down
56 changes: 53 additions & 3 deletions docs/md/explanation/view/advanced.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,61 @@ min_val, max_val = view.get_min_max("Sales")

</div>

## Describing a View Config

`Table::describe` validates a complete view config against a table and
reports the schema a `View` built from it would have - without creating one.
`describe` reports a `view_schema` if and only if `Table::view` with the same
config would succeed, and the two schemas are equal. It costs no engine
resources, so it is the right way to check a config before applying it.

<div class="javascript">

```javascript
const verdict = await table.describe({
columns: ["Sales", "margin"],
group_by: ["Region"],
expressions: { margin: '"Profit" / "Sales"' },
aggregates: { margin: "avg" },
});

if ("view_schema" in verdict) {
// { expression_schema: { margin: "float" },
// view_schema: { Sales: "float", margin: "float" } }
} else if ("expression_errors" in verdict) {
// { expression_schema: {...}, expression_errors: { margin: { error_message, line, column } } }
} else {
// { config_error: "Invalid column 'Sales' found in View columns." }
}
```

</div>
<div class="python">

```python
verdict = table.describe(
columns=["Sales", "margin"],
group_by=["Region"],
expressions={"margin": '"Profit" / "Sales"'},
aggregates={"margin": "avg"},
)

if "view_schema" in verdict:
...
elif "expression_errors" in verdict:
...
else:
verdict["config_error"]
```

</div>

## Expression Validation

Before creating a `View` with expressions, you can validate them against the
table's schema using `Table::validate_expressions`. This returns information
about which expressions are valid and their inferred types:
`Table::validate_expressions` is a specialization of `Table::describe` over
a config that selects no columns, so only the expressions are checked. It
returns which expressions are valid and their inferred types, plus an
`expression_alias` map echoing the request:

<div class="javascript">

Expand Down
Loading
Loading