docs: add version picker and manual release snapshots by yiaany · Pull Request #24792 · apache/datafusion

yiaany · 2026-08-30T09:16:54Z

Which issue does this PR close?

Closes Versioned documentation #17071.
Depends on [asf-site] docs: publish 55.0.0 and 55.1.0 documentation snapshots #25881
Depends on docs: preserve released documentation during deployment #25880

Rationale for this change

The documentation site currently follows main, so users cannot easily read the documentation that was published with a released DataFusion version. This change adds a version picker while keeping the site root on the current development documentation.

What changes are included in this PR?

Adds version switcher.

Are these changes tested?

Tested manually

Are there any user-facing changes?

Yes. The site will expose a version switcher and release URLs under /versions/<version>/, while the root URL continues to serve the latest main documentation.

yiaany · 2026-09-12T12:40:37Z

Hi @alamb and maintainers, just following up on this draft PR. It has been open since August 30, 2026, and the implementation and local validation are complete. The remaining question is the bootstrap sequence for publishing versions/55.0.0/ to asf-site. Could someone advise whether option 1 or option 2 in the PR description is preferred? I am happy to make any changes needed. Thanks!

alamb · 2026-09-13T20:56:11Z

Thank you @yiaany -- I had not seen this -- I will review it tomorrow

yiaany · 2026-09-28T10:47:29Z

Hi @alamb, just checking in on this when you have a chance. It has been about two weeks since your last message. The checks are still green and I have not changed the PR since then. Please let me know if you would like any changes or if there is anything I should prepare for the review. Thanks!

alamb · 2026-09-28T15:23:53Z

I am so sorry -- yes this is one of those important but not urgent PRs that I have let slip down. 😢

I will test it out day

alamb

Thank you @yiaany

I apologize for the delay in reviewing. I was struggling to find enough time to write a proper response.

At a high level, I think this PR is too complicated for what it is doing-- which is probably my fault for not defining it more specifically. I apologize for not being clear

I was hoping:

a static version list (you have it in versions.json I think)
documentation of how old versions are published as part of the release process

I don't think we need to change the current CI for publishing / building documentation

Then I was imagining we test it locally like:

manually build docs for a few versions (55.1.0 and 55.0.0 for example) and make a PR to the asf-site branch with the proposed layouts
Build the docs from this PR
Check out asf-site branch manually
move the manually built docs from the PR and put the versioned docs in the right plce

If that looks good we could merge this main PR and I think the main docs site would be updated.

I am not sure how much value all the various python test scripts add. If you think they are important, we should document clearly their intent and what types of regressions / breakages they protect again

Could a maintainer confirm which bootstrap procedure should be used?

I suggest Build older site versions one statically / part of the release process (not via a CI action)

Finally, this might be easier to test if you setup your fork so it published your forks asf-site branch as a github pages, and then test out the code / picker there. That would also make it easy for other reviewers to test it out / see what it looks like

alamb · 2026-09-28T20:36:10Z

    branches:
      - main
    paths:
+      - .gitattributes


i think these changes are unrelated to this PR (to trigger on changes to pyproject/uv) -- can you please make a separate PR to add them to main (it will be easier to review / merge) along with the rationale?

alamb · 2026-09-28T20:37:57Z

+# specific language governing permissions and limitations
+# under the License.
+
+"""Replace the current site while retaining immutable full-site snapshots."""


Why do we need to replace the current site? This is pretty confusing to me

alamb · 2026-09-28T20:38:54Z

+# specific language governing permissions and limitations
+# under the License.
+
+"""Build one complete documentation snapshot from an exact release tag."""


don't we already have a build.sh script to do this? It seems like a lot of python code to call a few scripts 🤔

alamb · 2026-09-28T20:39:35Z

+# specific language governing permissions and limitations
+# under the License.
+
+"""Validate current and immutable full-site documentation output."""


What needs to be validated? I don't undertstand why we need a 400 line python script here

alamb · 2026-09-28T20:43:36Z

@@ -0,0 +1,14 @@
+[


Yes, this looks good

alamb · 2026-09-28T20:50:07Z

 ## Dependencies

-Install build dependencies and build the documentation using
+From the repository root, install the documentation dependencies using


if this needs changing, perhaps you can make another standalone PR to update the build docs

alamb · 2026-09-28T20:50:33Z

 `docs/scripts/generate_dependency_graph.sh`, so ensure `cargo`, `cargo-depgraph`
 (`cargo install cargo-depgraph --version ^1.6 --locked`), and Graphviz `dot`
 (`brew install graphviz` or `sudo apt-get install -y graphviz`) are available.
+`.gitattributes` keeps documentation shell scripts LF-terminated so the same


this seems like a somewhat irrelevant detail 🤔

alamb · 2026-09-28T20:51:04Z

+
+Then open http://localhost:8000/.
+
+The public and assembled layouts are:


is this list necessary? It seems like it will just get out of date over time

alamb · 2026-09-28T20:51:21Z

+and the release catalog. It contains `Development` at the site root and records
+each release's semantic version, exact tag, and exact 40-character commit.
+
+The first snapshot is the lightweight tag `55.0.0`, which peels to


"peels to"?

alamb · 2026-09-28T20:52:05Z

@@ -1,4 +1,6 @@
 .github/ export-ignore
+docs/*.sh text eol=lf


why is this needed?

yiaany · 2026-09-29T17:39:04Z

Thank you @yiaany

I apologize for the delay in reviewing. I was struggling to find enough time to write a proper response.

At a high level, I think this PR is too complicated for what it is doing-- which is probably my fault for not defining it more specifically. I apologize for not being clear

I was hoping:

a static version list (you have it in versions.json I think)

documentation of how old versions are published as part of the release process

I don't think we need to change the current CI for publishing / building documentation

Then I was imagining we test it locally like:

manually build docs for a few versions (55.1.0 and 55.0.0 for example) and make a PR to the asf-site branch with the proposed layouts

Build the docs from this PR

Check out asf-site branch manually

move the manually built docs from the PR and put the versioned docs in the right plce

If that looks good we could merge this main PR and I think the main docs site would be updated.

I am not sure how much value all the various python test scripts add. If you think they are important, we should document clearly their intent and what types of regressions / breakages they protect again

Could a maintainer confirm which bootstrap procedure should be used?

I suggest Build older site versions one statically / part of the release process (not via a CI action)

Finally, this might be easier to test if you setup your fork so it published your forks asf-site branch as a github pages, and then test out the code / picker there. That would also make it easy for other reviewers to test it out / see what it looks like

Hi @alamb, thank you for the detailed review. You were right: I made the first version too complicated.

I’ve simplified #24792 to a static version list, the PyData version picker, and instructions for building and publishing release docs manually. I removed the custom build, deployment, validation, and test scripts. The existing docs CI and publishing workflow are unchanged in this PR.

I built the docs from the 55.0.0 and 55.1.0 tags and tested the version picker and missing-page fallback locally in a browser. The proposed generated layout is in #25881.

I also found that the current deployment’s rsync --delete would remove manually published release docs. I put the one-line fix in a separate PR, #25880. That change needs to land before the snapshots are published.

I haven’t set up a public GitHub Pages preview yet. Could you take another look at the smaller approach and let me know whether this matches what you had in mind?

…7071

alamb · 2026-10-08T21:09:35Z

Thank you @yiaany

I started going through this and made some changes / added documentation about how to test this locally.

I need to take one more pass through the instructions and then I think it will be good

martin-g · 2026-10-09T09:02:57Z

+
+**3. Add the release to the version picker.** Open a PR against `main` that
+adds the release to `docs/source/_static/versions.json`. Put the new release
+first and move `"preferred": true` to it, so that it is the default shown.


I see no "preferred": true currently at https://github.com/apache/datafusion/pull/24792/changes#diff-99ab87d9590045ea1e68507c78e72d2034bf0505d2b96edd222b3fe0e97fea1aR2-R6

- related to apache#17071 The documentation deploy uses rsync --delete, which would erase manually published versions/ directories. Exclude /versions/ so normal development-site deployments preserve release documentation. This one-line deployment change is intentionally separate from apache#24792. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…25881) - related to #17071 Publish complete generated documentation for the 55.0.0 and 55.1.0 release tags under versions/ on the asf-site branch. Built using the tagged docs/build.sh and the small Sphinx configuration overlay in #24792. The generated pages include a version picker, release-specific canonical URLs, search indexes and static assets. Verified locally with browser navigation across all three versions and the missing-page fallback. This branch contains only one publication commit on top of Apache asf-site; - merging should follow the archive-retention change in #25880. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

yiaany added 2 commits August 29, 2026 21:06

docs: add full-site version snapshots

e31a79f

docs: harden versioned docs publication

d7ac573

github-actions Bot added documentation Improvements or additions to documentation development-process Related to development process of DataFusion labels Aug 30, 2026

alamb reviewed Sep 28, 2026

View reviewed changes

docs: simplify versioned documentation workflow

d022e1e

yiaany changed the title ~~docs: add full-site version snapshots~~ docs: add version picker and manual release snapshots Sep 29, 2026

alamb added 9 commits October 8, 2026 16:20

Merge remote-tracking branch 'apache/main' into docs/versioned-docs-1…

9313453

…7071

trim unecssary docs, remove PR comment pretending to be documetnation

59b9cf6

simplify and document local publish workflow

a1fc011

Update versions

5a7ee52

fix urls

98afe60

remove index

6ac6765

update build readme

53f8bb3

Update release/conf.py script and docs

f22bba2

try and simplicy

acd7dba

martin-g reviewed Oct 9, 2026

View reviewed changes

Update dev/release/README.md

0baa248

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>


		Then open http://localhost:8000/.

		The public and assembled layouts are:

Conversation

yiaany commented Aug 30, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

yiaany commented Sep 12, 2026

Uh oh!

alamb commented Sep 13, 2026

Uh oh!

yiaany commented Sep 28, 2026

Uh oh!

alamb commented Sep 28, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiaany commented Sep 29, 2026

Uh oh!

alamb commented Oct 8, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yiaany commented Aug 30, 2026 •

edited by alamb

Loading