From 65658b48e1542d76555271bad3d8bbe1dd161761 Mon Sep 17 00:00:00 2001 From: Andrey Fedorov Date: Mon, 19 Oct 2026 23:31:30 -0400 Subject: [PATCH] docs: say that downloading from IDC buckets is free The docs said the buckets are public and need no credentials, but never that egress is free (the Google Public Data Program and AWS Open Data Sponsorship Program cover it). Worse, the always-on MCP instructions said "warn before any download", which an agent can read as a cost warning. State that downloads are free in the user guide, idc://guide, INSTRUCTIONS and the get_cohort_urls description, and say the size check is about transfer time and disk space. Co-Authored-By: Claude Opus 5.5 --- docs/user-guide.md | 14 +++++++++++--- src/idc_api/mcp/server.py | 16 +++++++++++----- 2 files changed, 22 insertions(+), 8 deletions(-) diff --git a/docs/user-guide.md b/docs/user-guide.md index 19ad3f9..49862d5 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -42,7 +42,8 @@ them — in NLST, about two thirds of the series are analysis results (IDC v25). original data only, or `only` for analysis results only. The main queryable table is **`index`** — one row per **series**, the unit you filter, count, -and download. IDC is large (~100+ TB total), so always check counts/size before downloading. +and download. IDC is large (~100+ TB total), so always check counts/size before downloading: +downloads are free (see [§4](#4-getting-the-data)), but transfer time and disk space are not. ### The query surfaces, and how they relate @@ -530,8 +531,15 @@ rationale) is in [deployment.md](../dev/deployment.md). **Get a manifest, then pull the files directly from S3/GCS.** All series URLs point at **public AWS S3 and GCS buckets — no credentials needed** — so the transfer never goes through the API server (the server never moves bytes; retrieval always means URLs/manifests), works -the same against the hosted or a local instance, and scales to whole collections. Two ways to -do it: +the same against the hosted or a local instance, and scales to whole collections. + +**Downloading is free.** IDC's buckets are hosted through the Google Public Data Program and +the AWS Open Data Sponsorship Program, which +[cover egress fees](https://learn.canceridc.dev/data/organization-of-data/files-and-metadata), +so you are never billed for pulling IDC data — whether it lands on a laptop, an on-premises +cluster, or a cloud VM. You need neither a cloud account nor a billing project. + +There are two ways to download: 1. **Whole collection or analysis result** — the simplest path; `cohort/manifest`'s download payload emits it for you when your filter is only `collection_id` values, or only diff --git a/src/idc_api/mcp/server.py b/src/idc_api/mcp/server.py index c7dd516..75c11e9 100644 --- a/src/idc_api/mcp/server.py +++ b/src/idc_api/mcp/server.py @@ -60,9 +60,10 @@ (non-imaging) attributes — staging, demographics, therapy — use list_clinical_tables, then get_clinical_table_schema / get_clinical_table to read the rows (or run_sql against `clinical.`). -3. IDC is large (100+ TB) — always report counts/size_TB and warn before any download. To get - data, use get_cohort_urls / the returned `idc` commands — direct S3/GCS transfer from public - buckets, no server involved. +3. IDC is large (100+ TB) — always report counts/size_TB and warn before any download, about + transfer time and disk space, not cost: downloads are free (no egress fees). To get data, use + get_cohort_urls / the returned `idc` commands — direct S3/GCS transfer from public buckets, + no server involved. 4. Trust `filters_applied`, not your intent: surface any `warnings` a result carries, and treat an empty `filters_applied` as the whole archive rather than a cohort. 5. For "what's new / what changed in release N", call get_release_changes — release history is @@ -471,7 +472,7 @@ def get_cohort_urls( both return s3:// URLs (GCS is reached via its S3-compatible endpoint, same bucket name for almost every series; this matches idc-index, and is why `idc download-from-manifest` only ever expects s3:// lines). Returns up to `limit` URLs (increase for full manifests). These - are anonymous public URLs — easiest is the `idc` CLI (handles either cloud); driving it + are anonymous public URLs, free to download (no egress fees) — easiest is the `idc` CLI (handles either cloud); driving it yourself, `s5cmd --no-sign-request` works directly for source=aws, and for source=gcs add `--endpoint-url https://storage.googleapis.com`. `terms`, `ranges` and `analysis_results` filter as in build_cohort. At least one filter predicate is required: unfiltered, this would @@ -571,7 +572,8 @@ def whats_new(version: str = "") -> str: not nested under one). An analysis result's series carry the `collection_id` of the images they analyze, so a `collection_id` filter returns original data *and* analysis results; original data has a NULL `analysis_result_id`. The main table is `index` (one row per *series*). IDC is -large (~100+ TB) — always check size before suggesting a download. +large (~100+ TB) — always check size before suggesting a download: downloading is free, but +transfer time and disk space are not. **The tools form a few families that build on each other:** - *Discovery* (`get_stats`, `list_collections`, `get_collection`, `list_analysis_results`, @@ -630,6 +632,10 @@ def whats_new(version: str = "") -> str: transfer directly from S3/GCS with no server involved, and work the same whether this server is hosted or local. This server never moves files itself; hand the user the commands/URLs to run. + *Downloading is free:* the buckets are hosted through the Google Public Data Program and + the AWS Open Data Sponsorship Program, which cover egress fees — the user is never billed, + needs no cloud account, and pays nothing whether the data goes to a laptop or a cloud VM. + If a user worries about cost, say so; the reason to check size first is time and disk. 5. *Be a good citizen:* check `get_licenses` (CC BY vs CC BY-NC) and, when publishing, include `get_citations` output — both the per-dataset `citations` and `idc_acknowledgment` (the IDC paper, https://doi.org/10.1148/rg.230180) to acknowledge IDC itself.