diff --git a/docs/user-guide.md b/docs/user-guide.md index 19ad3f9..49862d5 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -42,7 +42,8 @@ them — in NLST, about two thirds of the series are analysis results (IDC v25). original data only, or `only` for analysis results only. The main queryable table is **`index`** — one row per **series**, the unit you filter, count, -and download. IDC is large (~100+ TB total), so always check counts/size before downloading. +and download. IDC is large (~100+ TB total), so always check counts/size before downloading: +downloads are free (see [§4](#4-getting-the-data)), but transfer time and disk space are not. ### The query surfaces, and how they relate @@ -530,8 +531,15 @@ rationale) is in [deployment.md](../dev/deployment.md). **Get a manifest, then pull the files directly from S3/GCS.** All series URLs point at **public AWS S3 and GCS buckets — no credentials needed** — so the transfer never goes through the API server (the server never moves bytes; retrieval always means URLs/manifests), works -the same against the hosted or a local instance, and scales to whole collections. Two ways to -do it: +the same against the hosted or a local instance, and scales to whole collections. + +**Downloading is free.** IDC's buckets are hosted through the Google Public Data Program and +the AWS Open Data Sponsorship Program, which +[cover egress fees](https://learn.canceridc.dev/data/organization-of-data/files-and-metadata), +so you are never billed for pulling IDC data — whether it lands on a laptop, an on-premises +cluster, or a cloud VM. You need neither a cloud account nor a billing project. + +There are two ways to download: 1. **Whole collection or analysis result** — the simplest path; `cohort/manifest`'s download payload emits it for you when your filter is only `collection_id` values, or only diff --git a/src/idc_api/mcp/server.py b/src/idc_api/mcp/server.py index c7dd516..75c11e9 100644 --- a/src/idc_api/mcp/server.py +++ b/src/idc_api/mcp/server.py @@ -60,9 +60,10 @@ (non-imaging) attributes — staging, demographics, therapy — use list_clinical_tables, then get_clinical_table_schema / get_clinical_table to read the rows (or run_sql against `clinical.`). -3. IDC is large (100+ TB) — always report counts/size_TB and warn before any download. To get - data, use get_cohort_urls / the returned `idc` commands — direct S3/GCS transfer from public - buckets, no server involved. +3. IDC is large (100+ TB) — always report counts/size_TB and warn before any download, about + transfer time and disk space, not cost: downloads are free (no egress fees). To get data, use + get_cohort_urls / the returned `idc` commands — direct S3/GCS transfer from public buckets, + no server involved. 4. Trust `filters_applied`, not your intent: surface any `warnings` a result carries, and treat an empty `filters_applied` as the whole archive rather than a cohort. 5. For "what's new / what changed in release N", call get_release_changes — release history is @@ -471,7 +472,7 @@ def get_cohort_urls( both return s3:// URLs (GCS is reached via its S3-compatible endpoint, same bucket name for almost every series; this matches idc-index, and is why `idc download-from-manifest` only ever expects s3:// lines). Returns up to `limit` URLs (increase for full manifests). These - are anonymous public URLs — easiest is the `idc` CLI (handles either cloud); driving it + are anonymous public URLs, free to download (no egress fees) — easiest is the `idc` CLI (handles either cloud); driving it yourself, `s5cmd --no-sign-request` works directly for source=aws, and for source=gcs add `--endpoint-url https://storage.googleapis.com`. `terms`, `ranges` and `analysis_results` filter as in build_cohort. At least one filter predicate is required: unfiltered, this would @@ -571,7 +572,8 @@ def whats_new(version: str = "") -> str: not nested under one). An analysis result's series carry the `collection_id` of the images they analyze, so a `collection_id` filter returns original data *and* analysis results; original data has a NULL `analysis_result_id`. The main table is `index` (one row per *series*). IDC is -large (~100+ TB) — always check size before suggesting a download. +large (~100+ TB) — always check size before suggesting a download: downloading is free, but +transfer time and disk space are not. **The tools form a few families that build on each other:** - *Discovery* (`get_stats`, `list_collections`, `get_collection`, `list_analysis_results`, @@ -630,6 +632,10 @@ def whats_new(version: str = "") -> str: transfer directly from S3/GCS with no server involved, and work the same whether this server is hosted or local. This server never moves files itself; hand the user the commands/URLs to run. + *Downloading is free:* the buckets are hosted through the Google Public Data Program and + the AWS Open Data Sponsorship Program, which cover egress fees — the user is never billed, + needs no cloud account, and pays nothing whether the data goes to a laptop or a cloud VM. + If a user worries about cost, say so; the reason to check size first is time and disk. 5. *Be a good citizen:* check `get_licenses` (CC BY vs CC BY-NC) and, when publishing, include `get_citations` output — both the per-dataset `citations` and `idc_acknowledgment` (the IDC paper, https://doi.org/10.1148/rg.230180) to acknowledge IDC itself.