Repository navigation
fix(serve): set DJL HF cache defaults before _prepare_for_mode - #6435
Open
AkshayShah03 wants to merge 2 commits into
Open
AkshayShah03 wants to merge 2 commits into
AkshayShah03 wants to merge 2 commits into
Conversation
aws#5733 made the DJL builder default HF_HOME and HUGGINGFACE_HUB_CACHE to /tmp, because /opt/ml/model is read-only when artifacts are mounted there. The setdefault calls ran after _prepare_for_mode, which for SAGEMAKER_ENDPOINT merges the DJL server's default env (HF_HOME and HUGGINGFACE_HUB_CACHE = /opt/ml/model/) into env_vars with setdefault, so the /tmp defaults never took effect and the endpoint still cached HF downloads under the read-only /opt/ml/model. Apply the /tmp defaults before _prepare_for_mode. The existing tests mock _prepare_for_mode; the new test runs it for real (S3 upload mocked). Relates to aws#5698
AkshayShah03
had a problem deploying
to
manual-approval
October 12, 2026 17:57 — with 
GitHub Actions
Error
GitHub Actions
Error
AkshayShah03
had a problem deploying
to
manual-approval
October 12, 2026 17:57 — with
GitHub Actions
Error
GitHub Actions
Error
AkshayShah03
had a problem deploying
to
manual-approval
October 12, 2026 17:57 — with
GitHub Actions
Error
GitHub Actions
Error
The DJL server defaults set TRANSFORMERS_CACHE, HF_HOME and HUGGINGFACE_HUB_CACHE to /opt/ml/model/. The previous commit stops the read-only /opt/ml/model/ default from winning for HF_HOME and HUGGINGFACE_HUB_CACHE, but left TRANSFORMERS_CACHE at /opt/ml/model/. transformers uses TRANSFORMERS_CACHE as the default explicit cache_dir for from_pretrained, and an explicit cache_dir overrides HF_HUB_CACHE. So a container that loads through transformers still writes its Hub cache under the read-only /opt/ml/model/ even with HF_HOME and HUGGINGFACE_HUB_CACHE pointing at /tmp. That matches aws#5698, where the reporter set both of those by hand (which setdefault already honored) and still saw "OSError: [Errno 30] Read-only file system: /opt/ml/model/models--..." . Default TRANSFORMERS_CACHE to /tmp alongside the other two, with the same setdefault so a user-provided value still wins. Relates to aws#5698
mohamedzeidan2021
requested a deployment
to
manual-approval
October 23, 2026 23:19 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
mohamedzeidan2021
requested a deployment
to
manual-approval
October 23, 2026 23:19 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
mohamedzeidan2021
requested a deployment
to
manual-approval
October 23, 2026 23:19 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available: Relates to #5698
Description of changes:
#5733 made the DJL builder default
HF_HOMEandHUGGINGFACE_HUB_CACHEto/tmp, because/opt/ml/modelis read-only on the endpoint when artifacts are mounted there. Thesetdefaultcalls run after_prepare_for_mode, though. ForSAGEMAKER_ENDPOINT,_prepare_for_modemerges the DJL server's default env intoenv_varswithsetdefault:So the
/tmpdefaults never took effect, and the endpoint still caches Hub downloads under the read-only/opt/ml/model, the #5698 symptom. #5733's tests mock_prepare_for_mode, which hid this.This change applies the
/tmpdefaults before_prepare_for_mode. User-provided values still win because both places usesetdefault.TRANSFORMERS_CACHE(also/opt/ml/model/in the DJL defaults) is left as is, matching #5733's scope.Testing:
test_endpoint_build_hf_cache_is_tmp_with_real_prepare_for_moderuns the real_prepare_for_modewith only the S3 upload mocked. It fails before this change:HF_HOMEis/opt/ml/model/.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.