Repository navigation
Conversation
…atalog metadata
A hub model's task, license, provider, size, context window, languages,
modalities, model type, capabilities, and recipes live in
`HubContentSearchKeywords` as prefixed strings (`@task:`, `@license:`,
`@recipe:finetuning_sft_lora`, ...) and in the `HubContentDocument` JSON. The
SDK had no reader for them. `jumpstart/search.py`, `recipe_utils.py`, and
`training_session/_utils.py` each split the keywords for the one field they
needed, and the SageMaker agent skills repeat the full parse in four scripts.
The hub writes at most 50 keywords per model, so a model with many languages
loses its `@model-size:` and `@context-window:` keywords while the document
still carries the values. No consumer handled that.
Add `sagemaker.core.jumpstart.hub.catalog.HubModelCatalogEntry`, built from a
`HubContent` resource, a `DescribeHubContent` response, a `ListHubContents`
summary, or a `Search` record. A keyword is authoritative and the document
fills what the cap dropped. `HubRecipe` parses `@recipe:` keywords and
`RecipeCollection` entries into type, technique, and strategy, and the entry
answers `supports_technique("sft")` and `fine_tuning_techniques()` directly.
---
X-AI-Prompt: How do I read a JumpStart hub model's license, size, and supported fine-tuning techniques without parsing the search keywords myself?
X-AI-Tool: claude-code
|
Superseded by #6533, which combines the four changes into one PR and moves the hub-model metadata accessors onto |
There was a problem hiding this comment.
🤖 AI code review
This PR adds a new, self-contained HubModelCatalogEntry dataclass (plus HubRecipe and catalog_entries) under sagemaker.core.jumpstart.hub.catalog, giving callers a typed view of a hub model's catalog metadata read from HubContentSearchKeywords and the HubContentDocument, with keywords taking precedence and the document filling fields the 50-keyword cap dropped. The module is additive (no existing behavior changed) and comes with 9 focused unit tests covering keyword parsing, recipe parsing, document fill-in/precedence, malformed input, the resource path, and the list helper. I verified the required HubContent resource import, the shape attributes, and that the test construction matches existing fixtures; the implementation looks correct and well-tested. Only one minor readability note.
Reviewed commit f1a2150. Automated review; verify before acting.
| """Group keyword values by catalog attribute, in hub order.""" | ||
| values: Dict[str, List[str]] = {} | ||
| for keyword in keywords: | ||
| lowered = keyword.lower() |
There was a problem hiding this comment.
🟡 Low: This merged.extend(... if v.lower() not in _lower_set(merged)) relies on _lower_set(merged) being re-evaluated for each item while merged is simultaneously grown by extend. It is correct (and even dedups duplicates within document_hits), but the dependence on lazy generator evaluation against the list being mutated is subtle. Consider building the lowered set once and updating it explicitly in a plain loop so the dedup intent is obvious to future maintainers.
Problem
A hub model's task, license, provider, size, context window, languages, modalities, model type, capabilities, and recipes live in
HubContentSearchKeywordsas prefixed strings (@task:,@license:,@recipe:finetuning_sft_lora, ...) and in theHubContentDocumentJSON. The SDK has no reader for them.jumpstart/search.py,common_utils/recipe_utils.py, andtraining_session/_utils.pyeach split the keywords for the one field they use, and the SageMaker agent skills repeat the full parse in four scripts. The hub writes at most 50 keywords per model, so a model with many languages loses its@model-size:and@context-window:keywords while the document still carries the values. No consumer handled that, so a size filter over the keywords silently excluded those models.Solution
Add
sagemaker.core.jumpstart.hub.catalog.HubModelCatalogEntry, a dataclass built from aHubContentresource, aDescribeHubContentresponse, aListHubContentssummary, or aSearchrecord (from_hub_content, pluscatalog_entriesfor a list). It reads every keyword prefix into a typed field, keeps the hub's spelling, and lets the document fill only what the keyword cap dropped (a keyword is authoritative).HubRecipeparses@recipe:keywords andRecipeCollectionentries into type, technique, and strategy, so an entry answerssupports_technique("sft")andfine_tuning_techniques()without a second lookup.customization_supportedandbedrock_supportedread the@capability:keywords. The training guide gains a usage section.Tests
New
sagemaker-core/tests/unit/jumpstart/hub/test_catalog.py, 9 tests: every keyword prefix from a list summary, recipe parsing with and without a strategy, malformed recipe keywords ignored, the document filling fields the cap dropped, a keyword winning over the document for a scalar field, a malformed document giving an empty entry, the name requirement, aHubContentresource withUnassigned()fields, andcatalog_entriesover a list.The module is new, so
origin/masterhas no behavior to compare.black --line-length 100,flake8, andpydocstylepass on the new files.Merge Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.