Repository navigation
Save a lowered program whose named data is file-backed - #23693
Draft
shoumikhin wants to merge 1 commit into
Draft
shoumikhin wants to merge 1 commit into
shoumikhin wants to merge 1 commit into
Conversation
A backend can hand its named data to ExecuTorch as FileBackedData instead of
bytes, so a large blob waits on disk rather than as a second copy in memory.
The CUDA backend stores its weights this way. NamedDataStore, to_executorch()
and write_tensor_data_to_file() all handle it.
exir.save() of a program that contains such a lowered module fails, though:
TypeError: Object of type FileBackedData is not JSON serializable
The serializer turns the lowered module's named data into JSON with an
encoder that knows only bytes. Loading already reads every buffer back as
bytes.
The serializer now reads file-backed buffers into bytes before encoding them,
on a copy of the named data, so the lowered module keeps its file-backed
buffers. The saved file is the same as if the backend had given bytes. Like
any buffer in this JSON format, the data is held in memory while saving.
Tested with a new test in exir/tests/test_serde.py: it lowers a small graph
with the named-data test backend, replaces its buffer with a FileBackedData
holding distinct bytes, saves and loads the program, and checks the loaded
buffer. It fails without this change with the error above. All 16 tests in
that file pass with it.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23693
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 1 PendingAs of commit 1314b9f with merge base f2575be ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A backend can give ExecuTorch its named data as
FileBackedDatainstead ofbytes. The data then waits in a file on disk until the.pteor.ptdis written, instead of sitting in memory a second time. The CUDA backend stores its weights this way, andNamedDataStore,to_executorch()andwrite_tensor_data_to_file()all handle it.Saving such a lowered program with
exir.save()fails, though:The serializer writes the lowered module's named data as JSON, and its encoder only knows
bytes.Change
Before encoding, the serializer reads each file-backed buffer into
bytes. It works on a copy of the named data, so the lowered module itself keeps its file-backed buffers. The saved file is the same as if the backend had givenbytes, and, as for any buffer in this JSON format, the data is held in memory while saving. Loading did not need a change: it already decodes every buffer back tobytes.Test plan
A new test in
exir/tests/test_serde.pylowers a small graph with the named-data test backend, replaces its buffer with aFileBackedDataholding distinct bytes, then saves and loads the program and checks the loaded buffer. It fails without this change with the error above, and passes with it.Ran
exir/tests/test_serde.pyon Linux aarch64 with Python 3.12, using this branch'sexirpackage and schema over an installed nightly: 16 passed with the change, and 15 passed onmain(all tests except the new one).