Repository navigation
Eliminate S3 bucket access #112
Description
Once the mechanisms have switched over and enough traffic is on https://semanticsql.berkeleybop.io/{db}.gz , we will turn off public access to the underlying S3 bucket.
As well, we have now setup a logger on the bbop-sqlite bucket so that we can see who the stragglers are in more detail.
See: #110
Activity
kltm commentedon Jul 14, 2026
[bot] Comment posted by a Claude Code agent on behalf of @kltm.
Aggregate data from the bbop-sqlite access logger (~6 days, 2026-07-09 → 07-14), to help scope the straggler drain before removing public access:
semanticsql.berkeleybop.io) at 87–90% edge-cache-hit. The fronting works; consumers largely have not migrated yet.urllib(multiple 3.x versions) with 0% conditional GETs (every fetch is a full re-download), from hundreds of ephemeral IPs, ~half Microsoft/Azure — i.e. GitHub-Actions / cloud CI on OAK ≤0.7.1 (raw-S3 + urllib path). Matches the pre-0.7.2 code exactly.go.db.gzis the single largest, thenchebi,mondo,rhea,pr,ncit,hp,ncbitaxon— each repeat-pulled by hundreds of distinct IPs (exactly what the CDN's shared cache collapses).facebookexternalhitcrawler is pulling multi-GB.db.gzfiles — not a real consumer; blockable independent of this effort.Implication for lockdown: the raw URL still carries the majority with a live, identifiable OAK-ecosystem-CI consumer set, so removing
AllUsersnow would break them. The drain should accelerate as OAK ≥0.7.2 propagates — chiefly the ODK image bump (INCATools/ontology-development-kit#1354 / #1355) — and the direct hardcoders in #115 repoint. Suggest gating removal on this logger showing raw-URL object GETs fall off.— Posted by Claude Code agent on behalf of @kltm.
Let a dependency change trigger the id-label gate (#495)
Migrate off the raw S3 bucket for semantic-sql (#488) (#493)
Bump oaklib to >=0.7.3 so sqlite:obo: resolves against the CDN (#562) (…
kltm commentedon Aug 28, 2026
[bot] Posted by a Claude Code agent on behalf of @kltm.
Done — raw public access to
bbop-sqlitewas retired 2026-08-28 ~23:30 UTC. The bucket now has a full Public Access Block; CloudFront reaches it via Origin Access Control with a distribution-scoped policy grantings3:GetObjectands3:ListBucket.https://semanticsql.berkeleybop.iois the sole supported endpoint — object downloads and V1 root listing both verified end-to-end post-flip. Raws3.amazonaws.com/bbop-sqliteURLs and unsigned SDK/CLI access now return AccessDenied.Listing decision, as implemented: V1 root listing is preserved through the CDN (the ListBucket grant). SDK/ListObjectsV2 and
aws s3 lsagainst the bucket do not work — affected consumers (see #115) should read the root XML or await a manifest file if one is published later.Stragglers: denied raw requests are logged; anyone broken by this should swap to the CDN URL (drop-in, same paths) — or comment here.
— Posted by Claude Code agent on behalf of @kltm.
kltm commentedon Aug 28, 2026
This has been executed.
Fix retired S3 bucket: repoint downloads to the SemanticSQL CDN (#70)