feat(dataframe-storage): store a block's result automatically

[tkislan]

@coderabbitai ignore

Summary

Phase 2 of the "Full Dataframe storage" RFC (§3.6.4): storing a block's result automatically. Stacked on #135, which adds artifacts.store_dataframe and the writer; this PR only adds the hook on top of it, so review that one first. The diff against it is the hook, its tests and one manifest field.

When a Python or SQL block has a storage setting, the webapp puts the setting's frame name in the execute_request metadata (deepnote.dataframeStorage = {name, blockId}). A post_run_cell hook then stores the cell's result under that name, as if the block ended with artifacts.store_dataframe(result, name), and records the block in the manifest as block_id. A later call to store_dataframe writes no block_id, so a frame that code stored last belongs to no block.

• deepnote_toolkit/dataframe_storage.py: the hook, registered as one more guarded step in init_deepnote_runtime(), the metadata reader and the error reports.
• deepnote_toolkit/dataframe_storage_manifest.py: write_manifest takes an optional block_id.
• DEEPNOTE_DATAFRAME_STORAGE_ENABLED switches the hook on. It does not affect store_dataframe.
• No code is added to user cells.

Out of scope (other repos): the block setting, webapp and executor metadata, the feature flag, the read endpoint, and the cleanup that deletes a block's frame when its setting is cleared.

Behaviour worth reviewing

• Never raises, never prints into the cell. Warnings are suppressed. Failures are reported to the webapp once per kernel session per errno or exception type, as TOOLKIT_RUNTIME_ERROR with code: DATAFRAME_STORAGE_WRITE_FAILED (the only runtime type toolkit/errors accepts), with requests like the other userpod-API callers. An error status from the webapp is logged to the file logger as well. The report carries the exception type, never its message, because pyarrow quotes the failing value.
• Stop during a write. signal.default_int_handler is installed for the write, because a cell with top-level await runs post_run_cell under a handler that only queues the interrupt. The reply is already decided by then, so the hook sets result.error_in_exec and calls showtraceback(), which makes the executor stop its queue.
• Untrusted inputs. name and blockId from request metadata must match [A-Za-z0-9_-]{1,128}, because the name becomes a directory.
• Skipped: a read-only mount (checked before converting), SQL query previews, PySpark and pandas-on-Spark frames, non-DataFrame results, failed or interrupted cells, and helper executions. Helpers are recognised by result.info.store_history, not execution_count: IPython 9 sets execution_count even when no history is stored, so the RFC's earlier "no execution_count" premise only holds on IPython 8.
• Env var: enabled for 1, true, yes, on (case-insensitive), the same set as _to_bool in deepnote_core/config/loader.py.

Changes since the previous revision of this PR

The RFC moved from per-block storage to named frames, and its first part (the function) moved to #135:

• Frames are written to deepnote_dataframes/<name>/, not .deepnote/dataframes/<notebookId>/<blockId>/.
• The request metadata is {name, blockId}. enabled and notebookId are gone: the webapp adds the field only for executions that should store.
• The writer, manifest, compression, read-only check and project-root logic are in feat(artifacts): add store_dataframe and delete_dataframe #135 and unchanged except for the layout.

Not verified here

• Python 3.10-3.12 and 3.14, and pyarrow 16.1. Only 3.13 / pyarrow 24.0 / pandas 2.2.3 / polars 1.39 ran locally.
• A real s3fs mounter, and whether polars writing to a Python file object surfaces a failed upload (fallback: flush plus fsync).
• A real kernel on the top-level-await path. IPython 9.12 (what the lock pins for Python 3.14) was run locally against the new tests, in a scratch install, not through a real kernel.
• The project root copies set_notebook_path()'s precedence, which uses home_dir without a /work suffix. Confirm on a pod that it equals the project mount.
• The DEEPNOTE_DATAFRAME_STORAGE_ENABLED value comes from the sandbox manager, which appends a project's own variables after the system ones, so a project can override it (RFC §3.6.2). Not fixed here.

Test plan

• poetry run python -m pytest tests/unit/dataframe_storage tests/unit/test_runtime_initialization.py -p no:randomly: 229 passed (the 141 tests of feat(artifacts): add store_dataframe and delete_dataframe #135 and 88 for the hook)
• The same tests under IPython 9.12.0, installed to a scratch dir (the lock's 3.14 pin): 229 passed
• Broke the hook on purpose in 20 places (flag, name and block-id validation, helper-execution and failed-cell checks, SIGINT handling, report de-duplication, raise_for_status, warning suppression, read-only and EROFS handling, block_id, registration at startup); every break failed a test
• black, isort, flake8 clean on the touched files; mypy deepnote_toolkit/ clean
• Full tests/unit: 1512 passed, 4 skipped, 1 failed. The failure is test_redshift_dialect.py::test_redshift_distribution_matches_python_version: the venv has two redshift packages installed, and it fails identically on a clean origin/main.
• CI across the Python / pyarrow matrix
• Pod test: mount upload on close, project root

🤖 Generated with Claude Code

https://claude.ai/code/session_01B441SpUqRn7QTzcnWxjNa5

[github-actions]

📦 Python package built successfully!

• Version: 2.8.0.dev7+7edf8d1
• Wheel: deepnote_toolkit-2.8.0.dev7+7edf8d1-py3-none-any.whl
• Install:

  pip install "deepnote-toolkit @ https://deepnote-staging-runtime-artifactory.s3.amazonaws.com/deepnote-toolkit-packages/2.8.0.dev7%2B7edf8d1/deepnote_toolkit-2.8.0.dev7%2B7edf8d1-py3-none-any.whl"

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.06%. Comparing base (62a4d0e) to head (3b63336).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@                        Coverage Diff                         @@
##           feat/artifacts-store-dataframe     #131      +/-   ##
==================================================================
+ Coverage                           77.57%   78.06%   +0.49%     
==================================================================
  Files                                 117      117              
  Lines                                6663     6671       +8     
  Branches                              972      973       +1     
==================================================================
+ Hits                                 5169     5208      +39     
+ Misses                               1184     1153      -31     
  Partials                              310      310              

Flag	Coverage Δ
combined	78.06% <100.00%> (+0.49%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[deepnote-bot]

🚀 Review App Deployment Started

📝 Description	🌐 Link / Info
🌍 Review application	ra-131
🔑 Sign-in URL	Click to sign-in
📊 Application logs	View logs
🔄 Actions	Click to redeploy
🚀 ArgoCD deployment	View deployment
⏰ Last deployed	2026-10-08 13:13:35 (UTC)
📜 Deployed commit	9532db91e5983dfca365876da6a11c692486b6c9
🛠️ Toolkit version	7edf8d1