Skip to content

feat: expose the created job as latest_training_job and return it from ModelTrainer.train() - #6527

Closed
evakravi wants to merge 1 commit into
aws:masterfrom
evakravi:feat/trainer-latest-training-job
Closed

evakravi wants to merge 1 commit into
aws:masterfrom
evakravi:feat/trainer-latest-training-job

Conversation

@evakravi

Copy link
Copy Markdown
Member

Problem

ModelTrainer.train() returns None and stores the created job on the private _latest_training_job. A caller that wants the job name or ARN after train(wait=False) reads the private attribute or pages ListTrainingJobs by name prefix. The SDK's own docs do the former (docs/training/index.rst, docs/model_customization/model_customization.rst), seven v3-examples notebooks do the same, and ModelBuilder reads self.model._latest_training_job in three places. The SageMaker agent skills (aws/agent-toolkit-for-aws, aws-ai-ml/references/training-jobs) refuse to read a private attribute, so each of their three training templates polls list_training_jobs(NameContains=<prefix>) plus describe_training_job to recover what the trainer already holds.

DPOTrainer.train() also stored its job on latest_training_job while SFTTrainer, RLVRTrainer, and RLAIFTrainer used _latest_training_job, so show_metrics() and stream_logs(), which read the private name, could not find a DPO job.

Solution

Add a public latest_training_job property with a typed setter on ModelTrainer and BaseTrainer (the setter accepts a TrainingJob or None, and raises TypeError with a TrainingJob.get(<name>) hint for a bare string). ModelTrainer.train() returns the created TrainingJob for a SageMaker training job, after the wait when wait=True, and None for a dry run, a PipelineSession capture, and local mode. DPOTrainer stores its job on the shared attribute. The docs and the error messages of show_metrics(), stream_logs(), and ModelBuilder point at the public name. No existing attribute or return value changes for a caller that reads _latest_training_job.

Tests

New sagemaker-train/tests/unit/train/test_latest_training_job.py, 11 tests: train() returns the created job with and without wait, dry_run returns None and creates nothing, the property is None before train(), the setter attaches a job and rejects a string, output_data works through the setter, BaseTrainer round-trips the setter, show_metrics() and stream_logs() raise the new message, and SFTTrainer and DPOTrainer expose the created job after train(wait=False).

cd sagemaker-train && python -m pytest tests/unit/train/test_latest_training_job.py -q
11 passed

On origin/master the same file reports 9 failed, 2 passed (AttributeError: 'ModelTrainer' object has no attribute 'latest_training_job', assert None is <TrainingJob>).

Existing trainer suites on this branch: tests/unit/train/test_model_trainer.py, test_sft_trainer.py, test_dpo_trainer.py, test_rlvr_trainer.py, test_rlaif_trainer.py, test_base_trainer_serverful.py, test_base_trainer_compute.py, test_mtrl_eval_mlflow_url.py: 334 passed. sagemaker-serve tests/unit/test_model_builder_v3.py and test_model_builder_core.py: 63 passed, the one failure (test_build_with_vllm_for_text_generation, an S3 GetObject with expired credentials) fails identically on origin/master.

black --line-length 100, flake8, and pydocstyle pass on the changed files.

Merge Checklist

  • I read the contribution guide.
  • The change is backward compatible.
  • The commits use the repository commit-message format.
  • The change creates no S3 or STS client.
  • The change has regression coverage and adds no dependency.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…m ModelTrainer.train()

`ModelTrainer.train()` returned `None` and kept the created job on the private
`_latest_training_job`. A caller that needed the job name or ARN after
`train(wait=False)` had to read the private attribute or page
`ListTrainingJobs` by name prefix. The SDK's own docs, seven example
notebooks, and `ModelBuilder` read the private attribute too, and `DPOTrainer`
stored its job on a different attribute than the other serverless trainers, so
`show_metrics()` and `stream_logs()` could not find a DPO job.

Add a public `latest_training_job` property with a typed setter on
`ModelTrainer` and `BaseTrainer`, return the created `TrainingJob` from
`ModelTrainer.train()` for a SageMaker training job (`None` for a dry run, a
pipeline capture, and local mode), store the DPO job on the shared attribute,
and point the docs and error messages at the public name.

---
X-AI-Prompt: How do I get the training job that ModelTrainer.train() created without reading a private attribute?
X-AI-Tool: claude-code
@evakravi
evakravi deployed to auto-approve October 23, 2026 17:41 — with GitHub Actions Active
@evakravi
evakravi deployed to auto-approve October 23, 2026 17:42 — with GitHub Actions Active
@evakravi
evakravi deployed to auto-approve October 23, 2026 17:44 — with GitHub Actions Active
@evakravi

Copy link
Copy Markdown
Member Author

Superseded by #6533, which combines the four changes into one PR and moves the hub-model metadata accessors onto HubContent itself (a HubContentMixin in sagemaker.core.hub_content) instead of a separate class.

@evakravi evakravi closed this Oct 23, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

This PR adds a public latest_training_job property (with a typed setter that rejects bare strings) to ModelTrainer and BaseTrainer, makes ModelTrainer.train() return the created TrainingJob (and None for dry-run/PipelineSession/local mode), fixes DPOTrainer to store on the shared _latest_training_job, and updates docs and error messages to point at the public name. I checked the pydantic PrivateAttr backing on ModelTrainer, the ABC-based BaseTrainer, all subclass trainers, callers of _latest_training_job, and existing tests — the change is consistent and backward compatible for private-attr readers, and the new test file covers the behaviour. The change aligns with the convention from PR #6436 (store on _latest_training_job). No blocking issues found; one incidental typo fix is included.
[]

Reviewed commit bc92c8f. Automated review; verify before acting.

This branch was successfully deployed

1 active deployment
auto-approve — bc92c8f6 Deployed Oct 23, 2026 by evakravi via wait-for-approval #736
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant