Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions docs/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ them — in NLST, about two thirds of the series are analysis results (IDC v25).
original data only, or `only` for analysis results only.

The main queryable table is **`index`** — one row per **series**, the unit you filter, count,
and download. IDC is large (~100+ TB total), so always check counts/size before downloading.
and download. IDC is large (~100+ TB total), so always check counts/size before downloading:
downloads are free (see [§4](#4-getting-the-data)), but transfer time and disk space are not.

### The query surfaces, and how they relate

Expand Down Expand Up @@ -530,8 +531,15 @@ rationale) is in [deployment.md](../dev/deployment.md).
**Get a manifest, then pull the files directly from S3/GCS.** All series URLs point at
**public AWS S3 and GCS buckets — no credentials needed** — so the transfer never goes through
the API server (the server never moves bytes; retrieval always means URLs/manifests), works
the same against the hosted or a local instance, and scales to whole collections. Two ways to
do it:
the same against the hosted or a local instance, and scales to whole collections.

**Downloading is free.** IDC's buckets are hosted through the Google Public Data Program and
the AWS Open Data Sponsorship Program, which
[cover egress fees](https://learn.canceridc.dev/data/organization-of-data/files-and-metadata),
so you are never billed for pulling IDC data — whether it lands on a laptop, an on-premises
cluster, or a cloud VM. You need neither a cloud account nor a billing project.

There are two ways to download:

1. **Whole collection or analysis result** — the simplest path; `cohort/manifest`'s download
payload emits it for you when your filter is only `collection_id` values, or only
Expand Down
16 changes: 11 additions & 5 deletions src/idc_api/mcp/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,10 @@
(non-imaging) attributes — staging, demographics, therapy — use list_clinical_tables, then
get_clinical_table_schema / get_clinical_table to read the rows (or run_sql against
`clinical.<table>`).
3. IDC is large (100+ TB) — always report counts/size_TB and warn before any download. To get
data, use get_cohort_urls / the returned `idc` commands — direct S3/GCS transfer from public
buckets, no server involved.
3. IDC is large (100+ TB) — always report counts/size_TB and warn before any download, about
transfer time and disk space, not cost: downloads are free (no egress fees). To get data, use
get_cohort_urls / the returned `idc` commands — direct S3/GCS transfer from public buckets,
no server involved.
4. Trust `filters_applied`, not your intent: surface any `warnings` a result carries, and treat
an empty `filters_applied` as the whole archive rather than a cohort.
5. For "what's new / what changed in release N", call get_release_changes — release history is
Expand Down Expand Up @@ -471,7 +472,7 @@ def get_cohort_urls(
both return s3:// URLs (GCS is reached via its S3-compatible endpoint, same bucket name for
almost every series; this matches idc-index, and is why `idc download-from-manifest` only
ever expects s3:// lines). Returns up to `limit` URLs (increase for full manifests). These
are anonymous public URLs — easiest is the `idc` CLI (handles either cloud); driving it
are anonymous public URLs, free to download (no egress fees) — easiest is the `idc` CLI (handles either cloud); driving it
yourself, `s5cmd --no-sign-request` works directly for source=aws, and for source=gcs add
`--endpoint-url https://storage.googleapis.com`. `terms`, `ranges` and `analysis_results`
filter as in build_cohort. At least one filter predicate is required: unfiltered, this would
Expand Down Expand Up @@ -571,7 +572,8 @@ def whats_new(version: str = "") -> str:
not nested under one). An analysis result's series carry the `collection_id` of the images they
analyze, so a `collection_id` filter returns original data *and* analysis results; original
data has a NULL `analysis_result_id`. The main table is `index` (one row per *series*). IDC is
large (~100+ TB) — always check size before suggesting a download.
large (~100+ TB) — always check size before suggesting a download: downloading is free, but
transfer time and disk space are not.

**The tools form a few families that build on each other:**
- *Discovery* (`get_stats`, `list_collections`, `get_collection`, `list_analysis_results`,
Expand Down Expand Up @@ -630,6 +632,10 @@ def whats_new(version: str = "") -> str:
transfer directly from S3/GCS with no server involved, and work the same whether this
server is hosted or local.
This server never moves files itself; hand the user the commands/URLs to run.
*Downloading is free:* the buckets are hosted through the Google Public Data Program and
the AWS Open Data Sponsorship Program, which cover egress fees — the user is never billed,
needs no cloud account, and pays nothing whether the data goes to a laptop or a cloud VM.
If a user worries about cost, say so; the reason to check size first is time and disk.
5. *Be a good citizen:* check `get_licenses` (CC BY vs CC BY-NC) and, when publishing, include
`get_citations` output — both the per-dataset `citations` and `idc_acknowledgment` (the IDC
paper, https://doi.org/10.1148/rg.230180) to acknowledge IDC itself.
Expand Down
Loading