Repository navigation
Conversation
…m ModelTrainer.train() `ModelTrainer.train()` returned `None` and kept the created job on the private `_latest_training_job`. A caller that needed the job name or ARN after `train(wait=False)` had to read the private attribute or page `ListTrainingJobs` by name prefix. The SDK's own docs, seven example notebooks, and `ModelBuilder` read the private attribute too, and `DPOTrainer` stored its job on a different attribute than the other serverless trainers, so `show_metrics()` and `stream_logs()` could not find a DPO job. Add a public `latest_training_job` property with a typed setter on `ModelTrainer` and `BaseTrainer`, return the created `TrainingJob` from `ModelTrainer.train()` for a SageMaker training job (`None` for a dry run, a pipeline capture, and local mode), store the DPO job on the shared attribute, and point the docs and error messages at the public name. --- X-AI-Prompt: How do I get the training job that ModelTrainer.train() created without reading a private attribute? X-AI-Tool: claude-code
|
Superseded by #6533, which combines the four changes into one PR and moves the hub-model metadata accessors onto |
There was a problem hiding this comment.
🤖 AI code review
This PR adds a public latest_training_job property (with a typed setter that rejects bare strings) to ModelTrainer and BaseTrainer, makes ModelTrainer.train() return the created TrainingJob (and None for dry-run/PipelineSession/local mode), fixes DPOTrainer to store on the shared _latest_training_job, and updates docs and error messages to point at the public name. I checked the pydantic PrivateAttr backing on ModelTrainer, the ABC-based BaseTrainer, all subclass trainers, callers of _latest_training_job, and existing tests — the change is consistent and backward compatible for private-attr readers, and the new test file covers the behaviour. The change aligns with the convention from PR #6436 (store on _latest_training_job). No blocking issues found; one incidental typo fix is included.
[]
Reviewed commit bc92c8f. Automated review; verify before acting.
Problem
ModelTrainer.train()returnsNoneand stores the created job on the private_latest_training_job. A caller that wants the job name or ARN aftertrain(wait=False)reads the private attribute or pagesListTrainingJobsby name prefix. The SDK's own docs do the former (docs/training/index.rst,docs/model_customization/model_customization.rst), sevenv3-examplesnotebooks do the same, andModelBuilderreadsself.model._latest_training_jobin three places. The SageMaker agent skills (aws/agent-toolkit-for-aws,aws-ai-ml/references/training-jobs) refuse to read a private attribute, so each of their three training templates pollslist_training_jobs(NameContains=<prefix>)plusdescribe_training_jobto recover what the trainer already holds.DPOTrainer.train()also stored its job onlatest_training_jobwhileSFTTrainer,RLVRTrainer, andRLAIFTrainerused_latest_training_job, soshow_metrics()andstream_logs(), which read the private name, could not find a DPO job.Solution
Add a public
latest_training_jobproperty with a typed setter onModelTrainerandBaseTrainer(the setter accepts aTrainingJoborNone, and raisesTypeErrorwith aTrainingJob.get(<name>)hint for a bare string).ModelTrainer.train()returns the createdTrainingJobfor a SageMaker training job, after the wait whenwait=True, andNonefor a dry run, aPipelineSessioncapture, and local mode.DPOTrainerstores its job on the shared attribute. The docs and the error messages ofshow_metrics(),stream_logs(), andModelBuilderpoint at the public name. No existing attribute or return value changes for a caller that reads_latest_training_job.Tests
New
sagemaker-train/tests/unit/train/test_latest_training_job.py, 11 tests:train()returns the created job with and withoutwait,dry_runreturnsNoneand creates nothing, the property isNonebeforetrain(), the setter attaches a job and rejects a string,output_dataworks through the setter,BaseTrainerround-trips the setter,show_metrics()andstream_logs()raise the new message, andSFTTrainerandDPOTrainerexpose the created job aftertrain(wait=False).On
origin/masterthe same file reports9 failed, 2 passed(AttributeError: 'ModelTrainer' object has no attribute 'latest_training_job',assert None is <TrainingJob>).Existing trainer suites on this branch:
tests/unit/train/test_model_trainer.py,test_sft_trainer.py,test_dpo_trainer.py,test_rlvr_trainer.py,test_rlaif_trainer.py,test_base_trainer_serverful.py,test_base_trainer_compute.py,test_mtrl_eval_mlflow_url.py:334 passed.sagemaker-servetests/unit/test_model_builder_v3.pyandtest_model_builder_core.py:63 passed, the one failure (test_build_with_vllm_for_text_generation, an S3GetObjectwith expired credentials) fails identically onorigin/master.black --line-length 100,flake8, andpydocstylepass on the changed files.Merge Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.