From 9b3dbd06aca381f24216612941c2868131cc8081 Mon Sep 17 00:00:00 2001 From: SS-JIA Date: Sat, 10 Oct 2026 11:38:03 -0400 Subject: [PATCH] Fix Samsung YOLO26 compilation and bound model CI runtime MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Summary: ## Ulterior Motive Unblock viable/strict by fixing Samsung's model failure and multi-hour CI job. ## Rationale Expand scalar multiply constants for the SDK exporter; keep unsupported TopK and its output Q/DQ portable. Reduce devicefarm round trips, bound subprocesses, and use deterministic CI-only training budgets. Preserve example defaults, model coverage, and numerical tolerances. ## Details - Compilation - **Before:** Scalar attention multiplies fail SDK export. SDK v1.2.0 cannot export TopK; isolated output Q/DQ delegation can produce empty graphs. - **After:** Expand node-local scalar constants without mutating exported parameters; support real and fake tensors. Keep TopK and dependent output Q/DQ portable while delegating supported surrounding operators. Build/link Android quantized kernels and build/load host AOT schemas. The original TopK builder is unchanged. - Device execution - **Before:** Inference uses 12–13 unbounded devicefarm commands. - **After:** Bundle runner/model/inputs and use four commands. Each command has a 300-second deadline; timeout kills and reaps its owned process group. Validate inputs before device I/O. - Training - **Before:** CI fine-tunes full datasets, dominating runtime. - **After:** Use 256 FP16 samples and 128 pre-QAT samples, learning rate 5e-4, and isolated deterministic initialization. Preserve five FP16 epochs, one pre-QAT epoch, QAT updates, full validation/calibration, and atol=0.008/0.03. Optional sample/rate arguments leave example defaults unchanged. - Regression gates - **Before:** Expensive training can precede compiler failures; explicitly invoked test modules are absent from wheels. - **After:** Run host regressions and fail-fast primitive/YOLO26/scalar/TopK checks first. Retain four CI entry-point modules in wheels. Cover scalar shapes/fake tensors, portable quantized serialization, process-group deadlines, transfer counts, and training defaults/budgets/RNG isolation. Authored with OpenAI Codex. Test Plan: ## Observed Results Native results below validate the pre-rebase PR revision. This commit squashes that same net patch onto newer public main; native CI on the rebased revision must rerun. | Validation | Platform | Observed result | | --- | --- | --- | | Samsung native unit suite | Linux x86_64, SDK v1.2.0, E9955 CI | 20 passed, 0 failed, 0 skipped; 5.500 seconds | | Early compiler/device gate | Same CI | Nine methods passed, 0 failed, 0 skipped; 486.202 seconds. Eight primitive subcases and full YOLO26 compile; scalar and seven float/quantized TopK device comparisons passed | | Full Samsung model suite | Same CI | 14 passed, 0 failed, 0 skipped; 2473.714 seconds. Device acquired; FP16 atol=0.008 and QAT atol=0.03 unchanged | | Quantized model compilation | Linux/SDK | Passed; host AOT library built/deployed, Android portable kernels linked | | General Linux unit CI | Linux | 3174 Python passed, 107 skipped, 1 xfailed; all 100 C++ tests passed | | Complete pre-rebase PR CI | GitHub Actions | 398 checks passed, 23 expected skips, 0 failed, 0 pending | | Rebased host regressions | macOS arm64, Python 3.11 | 20 passed, 0 failed, 0 skipped; 3.916 seconds; SDK-only imports stubbed, real quantized schemas/serialization and parent/child timeout | | Rebased wheel retention | Same host | 21 passed, 0 failed, 0 skipped; 33.12 seconds; two optional pytest configuration warnings | | Rebased lint/shell/whitespace | Same host | Passed CMAKEFORMAT, FLAKE8, UFMT, LICENSELINT, NEWLINE, TORCH_AO_IMPORT; bash syntax and diff whitespace clean | | Squash/rebase patch preservation | Git | One commit above latest main; complete binary diff identical to pre-rebase net patch; no conflicts | | Rebased native SDK/device CI | Linux/E9955 | Not run at commit time; checks rerun after push | | YOLO26 device numerical comparison | E9955 | Not run: YOLO26 coverage is compilation-only | | Performance validation | Before | After | Scope | | --- | --- | --- | --- | | Samsung full model suite | 13 tests in 11824.509 seconds; YOLO26 failed | All 14 passed in 2473.714 seconds; 79.1% shorter | Historical SDK v1.2.0/E9955 CI comparison | | Whole Samsung job | 204.85 minutes | 59.9 minutes; 70.8% shorter | Historical comparison, includes new gates | | Mocked two-input inference | 13 commands | Four commands | Host regression; not wall-time measurement | | FP16 training workload | 4325 optimizer steps | 160 steps | Five epochs retained | | Pre-QAT training workload | 1065 steps | 64 steps | QAT workload unchanged | | Controlled A/B vs parent/latest public ancestor | Not run | Not run | No controlled paired device runs; historical timing is not same-parent A/B | ## Reproduction Guide ### Retrieve source and host dependencies Install Git, GitHub CLI, Python 3.11, CMake, and a C++20-capable compiler. Authenticate GitHub CLI when required: ```bash gh auth login gh repo clone pytorch/executorch executorch-samsung-ci cd executorch-samsung-ci gh pr checkout 23650 git submodule update --init --recursive python3.11 -m venv .venv source .venv/bin/activate ./install_executorch.sh python -m pip install evaluate 'accelerate>=1.1.0' pytest lintrunner -r requirements-lintrunner.txt -r examples/models/yolo26/requirements.txt cmake -S . -B build-quantized-aot -DCMAKE_BUILD_TYPE=Release -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=ON cmake --build build-quantized-aot --target quantized_ops_aot_lib -j 2 cp build-quantized-aot/kernels/quantized/libquantized_ops_aot_lib.* kernels/quantized/ ``` The library extension is .so on Linux and .dylib on macOS. On macOS, install standalone flatc if the bundled executable cannot run: ```bash brew install flatbuffers flatc --version ``` ### Host correctness and packaging Tests generate graphs and data. No pretrained files or Samsung device are required. SDK-only imports and external device I/O are mocked; quantized schemas, serialization, and timeout parent/child processes are real. A compatible local quantized AOT library and standalone flatc were used for macOS validation. ```bash python -u - <<'PY' import shutil import types import unittest from pathlib import Path from unittest.mock import patch import executorch executorch.__path__ = [str(Path.cwd())] modules = {name: types.ModuleType(name) for name in ('executorch.backends.samsung.python', 'executorch.backends.samsung.python.PyGraphWrapperAdaptor', 'executorch.backends.samsung.python.PyEnnWrapperAdaptor')} with patch.dict('sys.modules', modules): from executorch.exir._serialize import _flatbuffer if standalone_flatc := shutil.which('flatc'): _flatbuffer._flatc_cached_path = standalone_flatc suite = unittest.defaultTestLoader.discover('backends/samsung/test/unit') result = unittest.TextTestRunner(verbosity=2).run(suite) raise SystemExit(not result.wasSuccessful()) PY python -m pytest -q -o addopts= .ci/scripts/tests/test_wheel_test_modules.py ``` Optional native-extension/semaphore diagnostics appeared on the host; tests still exited successfully. Packaging emitted two optional pytest configuration warnings. ### Formatting and syntax ```bash lintrunner --take CMAKEFORMAT,FLAKE8,UFMT,LICENSELINT,NEWLINE,TORCH_AO_IMPORT $(git diff --name-only origin/main...HEAD) bash -n .ci/scripts/test-samsung-models.sh backends/samsung/build.sh git diff --check origin/main...HEAD ``` ### Native end-to-end correctness Upstream workflow-dispatch permission is required. CI provisions SDK v1.2.0, compiler, Android runner, credentials, checkpoints, datasets, and E9955 access; no author-local artifacts are required. The Samsung child script installs remaining model dependencies and runs all regression/model gates. ```bash gh workflow run trunk.yml --repo pytorch/executorch --ref codex/samsung-ci-runtime-yolo gh run list --repo pytorch/executorch --workflow trunk.yml --branch codex/samsung-ci-runtime-yolo gh run watch RUN_ID --repo pytorch/executorch --exit-status gh run view RUN_ID --repo pytorch/executorch --json jobs gh api repos/pytorch/executorch/actions/jobs/JOB_ID/logs gh pr checks 23650 --repo pytorch/executorch ``` Replace RUN_ID and JOB_ID with IDs printed by the preceding commands. Require 20 native unit tests, nine early-gate methods, and 14 model tests. Confirm actual device acquisition, not the existing unavailable-device skip. Keep FP16/QAT tolerances unchanged. YOLO26 must compile; scalar/TopK and existing full-model numerical comparisons must pass. Retrieve the successful pre-rebase logs and Linux unit results: ```bash gh api repos/pytorch/executorch/actions/jobs/114033451431/logs gh api repos/pytorch/executorch/actions/jobs/114033451063/logs gh api repos/pytorch/executorch/actions/jobs/114033473601/logs ``` ### Historical runtime comparison Retrieve baseline and successful PR metadata/logs. Compare model-suite `Ran ... tests in ... seconds` totals and job start/completion timestamps; these are historical, not controlled A/B runs. ```bash gh api repos/pytorch/executorch/actions/jobs/113816027745/logs gh api repos/pytorch/executorch/actions/jobs/114033451431/logs gh api repos/pytorch/executorch/actions/jobs/113816027745 --jq '{started_at, completed_at, conclusion}' gh api repos/pytorch/executorch/actions/jobs/114033451431 --jq '{started_at, completed_at, conclusion}' ``` Public baseline: https://github.com/pytorch/executorch/actions/runs/37929317498/job/113816027745 Successful pre-rebase run: https://github.com/pytorch/executorch/actions/runs/37993457812/job/114033451431 Reviewers: --- .ci/scripts/test-samsung-models.sh | 22 ++ .github/workflows/trunk.yml | 11 +- backends/samsung/build.sh | 4 + backends/samsung/builders/op_mul.py | 32 ++- backends/samsung/partition/enn_partitioner.py | 17 ++ backends/samsung/quantizer/quantizer.py | 1 + .../test/models/test_mobilebert_finetuning.py | 37 +-- .../test/models/test_mobilebert_qat.py | 4 +- backends/samsung/test/models/test_yolo26.py | 13 +- .../test/models/test_yolo26_primitives.py | 74 ++++++ backends/samsung/test/ops/test_mul.py | 4 + backends/samsung/test/ops/test_topk.py | 81 ++++++- .../test/unit/test_mobilebert_finetune.py | 174 ++++++++++++++ backends/samsung/test/unit/test_mul.py | 61 +++++ .../test/unit/test_runtime_executor.py | 134 +++++++++++ backends/samsung/test/unit/test_topk.py | 217 ++++++++++++++++++ .../samsung/test/utils/runtime_executor.py | 85 ++++--- examples/samsung/CMakeLists.txt | 10 +- .../samsung/scripts/mobilebert_finetune.py | 12 +- .../scripts/mobilebert_finetune_QAT.py | 17 +- setup.py | 4 + 21 files changed, 930 insertions(+), 84 deletions(-) create mode 100644 .ci/scripts/test-samsung-models.sh create mode 100644 backends/samsung/test/models/test_yolo26_primitives.py create mode 100644 backends/samsung/test/unit/test_mobilebert_finetune.py create mode 100644 backends/samsung/test/unit/test_mul.py create mode 100644 backends/samsung/test/unit/test_runtime_executor.py create mode 100644 backends/samsung/test/unit/test_topk.py diff --git a/.ci/scripts/test-samsung-models.sh b/.ci/scripts/test-samsung-models.sh new file mode 100644 index 00000000000..236799c7df6 --- /dev/null +++ b/.ci/scripts/test-samsung-models.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +set -ex + +pip install evaluate 'accelerate>=1.1.0' +pip install --upgrade-strategy only-if-needed \ + --extra-index-url https://download.pytorch.org/whl/cpu \ + -r examples/models/yolo26/requirements.txt + +python -m unittest discover -v -s backends/samsung/test/unit -p "test_*.py" +# Fail fast on compiler regressions before running fine-tuning tests. +python -m unittest -fv \ + executorch.backends.samsung.test.models.test_yolo26_primitives \ + executorch.backends.samsung.test.models.test_yolo26 \ + executorch.backends.samsung.test.ops.test_mul.TestMul.test_fp32_attention_scale \ + executorch.backends.samsung.test.ops.test_topk +python -m unittest discover -fv -s backends/samsung/test/models -p "test_*.py" diff --git a/.github/workflows/trunk.yml b/.github/workflows/trunk.yml index 02914548339..ad3426f48aa 100644 --- a/.github/workflows/trunk.yml +++ b/.github/workflows/trunk.yml @@ -1148,13 +1148,4 @@ jobs: exit 0 fi - # Test models - # TODO: Switch to run_tests.py once CI device provisioning is stable - # python -m executorch.backends.samsung.test.utils.run_tests --chipset E9955 - # accelerate is for test_mobilebert_qat, whose TrainingArguments needs it; - # transformers reports the same error when it is absent and when it is too old. - pip install evaluate 'accelerate>=1.1.0' - pip install --upgrade-strategy only-if-needed \ - --extra-index-url https://download.pytorch.org/whl/cpu \ - -r examples/models/yolo26/requirements.txt - python -m unittest discover -s backends/samsung/test/models -p "test_*.py" + bash .ci/scripts/test-samsung-models.sh diff --git a/backends/samsung/build.sh b/backends/samsung/build.sh index e6258152785..28c3dfbabd7 100755 --- a/backends/samsung/build.sh +++ b/backends/samsung/build.sh @@ -40,6 +40,8 @@ function build_x86_64() { -DCMAKE_INSTALL_PREFIX=${X86_64_BUILD_DIR} \ -DEXYNOS_AI_LITECORE_ROOT=${EXYNOS_AI_LITECORE_ROOT} \ -DEXECUTORCH_BUILD_ENN=ON \ + -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ + -DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=ON \ -DEXECUTORCH_BUILD_DEVTOOLS=ON \ -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \ -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \ @@ -54,6 +56,7 @@ function build_x86_64() { rm -f ${PROJECT_DIR}/backends/samsung/python/Py*so cp -fv ${X86_64_BUILD_DIR}/backends/samsung/Py*so ${PROJECT_DIR}/backends/samsung/python/ + cp -fv ${X86_64_BUILD_DIR}/kernels/quantized/libquantized_ops_aot_lib.so ${PROJECT_DIR}/kernels/quantized/ cp -fv ${PROJECT_DIR}/schema/*.fbs ${PROJECT_DIR}/exir/_serialize/ } @@ -85,6 +88,7 @@ function build_android() { -DCMAKE_BUILD_TYPE=Release \ -DEXECUTORCH_BUILD_ENN=ON \ -DEXYNOS_AI_LITECORE_ROOT=${EXYNOS_AI_LITECORE_ROOT} \ + -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \ -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \ -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \ diff --git a/backends/samsung/builders/op_mul.py b/backends/samsung/builders/op_mul.py index e703ddff253..ee0648aea8c 100644 --- a/backends/samsung/builders/op_mul.py +++ b/backends/samsung/builders/op_mul.py @@ -11,7 +11,9 @@ NodeVisitor, register_node_visitor, ) +from executorch.backends.samsung.builders.utils import get_map_dtype, get_tensor from executorch.backends.samsung.serialization.enn_graph_schema import EnnGraph +from executorch.backends.transforms.utils import is_param_node @register_node_visitor @@ -28,18 +30,32 @@ def define_node( vals_to_ids: Dict[torch.Tensor, int], ) -> bool: - input1 = node.args[0] - input_id_1 = self.define_tensor(input1, enn_graph, vals_to_ids) - - input2 = node.args[1] - input_id_2 = self.define_tensor(input2, enn_graph, vals_to_ids) + input_ids = [] + for input_node in node.args[:2]: + tensor = get_tensor(self.exported_program, input_node) + if is_param_node(self.exported_program, input_node) and tensor.dim() == 0: + # The SDK's multiply exporter cannot broadcast scalar constants. + tensor = tensor.expand(node.meta["val"].shape).contiguous() + const_data = None + if not isinstance(tensor, torch._subclasses.fake_tensor.FakeTensor): + const_data = tensor.cpu().detach().numpy() + input_ids.append( + enn_graph.define_tensor( + f"{node.name}_{input_node.name}", + list(tensor.shape), + get_map_dtype(tensor.dtype), + "CONSTANT", + const_data, + quant_param=input_node.meta.get("quantize_attrs"), + ) + ) + else: + input_ids.append(self.define_tensor(input_node, enn_graph, vals_to_ids)) params = {} self._update_params_qdtype(node, params) output_id = self.define_tensor(node, enn_graph, vals_to_ids) - enn_graph.define_op( - node.name, "ELTMUL", [input_id_1, input_id_2], [output_id], params - ) + enn_graph.define_op(node.name, "ELTMUL", input_ids, [output_id], params) return True diff --git a/backends/samsung/partition/enn_partitioner.py b/backends/samsung/partition/enn_partitioner.py index 6541f03aad9..2c35d40c030 100644 --- a/backends/samsung/partition/enn_partitioner.py +++ b/backends/samsung/partition/enn_partitioner.py @@ -4,6 +4,7 @@ # This source code is licensed under the BSD-style license found in the # LICENSE file in the root directory of this source tree. import logging +import operator from typing import Any, Callable, Dict, List, Optional, Tuple import executorch.backends.samsung.builders.node_visitor as node_visitor @@ -22,6 +23,7 @@ ENN_COMPILE_OPTION_TITLE, ) from executorch.backends.samsung.serialization.enn_graph_schema import EnnGraph +from executorch.backends.samsung.utils.constants import QuantConstants from executorch.backends.samsung.utils.utils import get_compile_spec from executorch.exir.backend.backend_details import CompileSpec from executorch.exir.backend.canonical_partitioners.pattern_op_partitioner import ( @@ -74,6 +76,18 @@ def is_node_supported(self, _, node: torch.fx.Node) -> bool: if node.op != "call_function": return False + # The SDK has no TopK exporter. Keep its output Q/DQ on CPU too, + # since Q/DQ-only delegate partitions disappear during preprocessing. + producer = node + while producer.target in ( + operator.getitem, + *QuantConstants.QUANT_OPS_KEY_MAP, + *QuantConstants.DEQUANT_OPS_KEY_MAP, + ): + producer = producer.args[0] + if producer.target == exir_ops.edge.aten.topk.default: + return False + if node.op in [ "get_attr", "placeholder", @@ -84,6 +98,9 @@ def is_node_supported(self, _, node: torch.fx.Node) -> bool: if node.target in SUPPORTED_OPS: return True + if node.target == operator.getitem: + return self.is_node_supported(_, node.args[0]) + if node.target.__name__ in self.node_visitors: enn_graph = EnnGraph() vals_to_ids: Dict[torch.fx.Node, int] = {} diff --git a/backends/samsung/quantizer/quantizer.py b/backends/samsung/quantizer/quantizer.py index 5f8df32e095..57f1be9ba28 100644 --- a/backends/samsung/quantizer/quantizer.py +++ b/backends/samsung/quantizer/quantizer.py @@ -6,6 +6,7 @@ from typing import Callable, Sequence +import executorch.kernels.quantized # noqa: F401 import torch from executorch.backends.samsung._passes.enn_pass_manager import EnnPassManager from torch.fx import GraphModule diff --git a/backends/samsung/test/models/test_mobilebert_finetuning.py b/backends/samsung/test/models/test_mobilebert_finetuning.py index 0c695f14045..08057f8ca5d 100644 --- a/backends/samsung/test/models/test_mobilebert_finetuning.py +++ b/backends/samsung/test/models/test_mobilebert_finetuning.py @@ -7,6 +7,8 @@ import unittest +import torch + from executorch.backends.samsung.serialization.compile_options import ( gen_samsung_backend_compile_spec, ) @@ -17,20 +19,23 @@ class Test_Milestone_MobileBertFinetune(unittest.TestCase): def test_mobilebert_finetuning_fp16(self): - mobilebert_finetune = MobileBertFinetune() - # Smaller batches, because training keeps every intermediate value until - # the backward pass is done and a test runner cannot hold them all. The - # number of passes is left alone: it is what the example script uses, and - # the range this model ends up with does not fall off with fewer of them. - model, _ = mobilebert_finetune.get_finetune_mobilebert(None, batch_size=8) - example_input = mobilebert_finetune.get_example_inputs() - tester = SamsungTester( - model, example_input, [gen_samsung_backend_compile_spec(TestConfig.chipset)] - ) + with torch.random.fork_rng(): + torch.manual_seed(8) + mobilebert_finetune = MobileBertFinetune() + # Exercise all five fine-tuning epochs on a bounded CI dataset. + model, _ = mobilebert_finetune.get_finetune_mobilebert( + None, batch_size=8, max_train_samples=256, learning_rate=5e-4 + ) + example_input = mobilebert_finetune.get_example_inputs() + tester = SamsungTester( + model, + example_input, + [gen_samsung_backend_compile_spec(TestConfig.chipset)], + ) - ( - tester.export() - .to_edge_transform_and_lower() - .to_executorch() - .run_method_and_compare_outputs(inputs=example_input, atol=0.008) - ) + ( + tester.export() + .to_edge_transform_and_lower() + .to_executorch() + .run_method_and_compare_outputs(inputs=example_input, atol=0.008) + ) diff --git a/backends/samsung/test/models/test_mobilebert_qat.py b/backends/samsung/test/models/test_mobilebert_qat.py index 5dc2ddbc538..bb472505a54 100644 --- a/backends/samsung/test/models/test_mobilebert_qat.py +++ b/backends/samsung/test/models/test_mobilebert_qat.py @@ -42,6 +42,8 @@ def setUp(self): self._random_state = random.getstate() self._np_random_state = np.random.get_state() self._torch_rng_state = torch.get_rng_state() + # Match Trainer's seed before initializing the classification head. + torch.manual_seed(42) def tearDown(self): random.setstate(self._random_state) @@ -67,7 +69,7 @@ def test_mobilebert_qat_a8w8(self): # Get the finetune model mobilebert_finetune = MobileBertFinetune(metric, args) model, tokenized_datasets = mobilebert_finetune.get_finetune_mobilebert( - args.artifact + args.artifact, max_train_samples=128, learning_rate=5e-4 ) # Configure QAT parameters diff --git a/backends/samsung/test/models/test_yolo26.py b/backends/samsung/test/models/test_yolo26.py index 2502d4d1a29..968c08c521a 100644 --- a/backends/samsung/test/models/test_yolo26.py +++ b/backends/samsung/test/models/test_yolo26.py @@ -19,9 +19,16 @@ class TestMilestoneYolo26(unittest.TestCase): def test_yolo26_fp16(self): torch.manual_seed(8) - model = YOLO26Model().get_eager_model() - example_input = YOLO26Model().get_example_inputs() + provider = YOLO26Model() + model = provider.get_eager_model() + example_input = provider.get_example_inputs() tester = SamsungTester( model, example_input, [gen_samsung_backend_compile_spec(TestConfig.chipset)] ) - (tester.export().to_edge_transform_and_lower().to_executorch()) + ( + tester.export() + .to_edge_transform_and_lower() + .check(["torch.ops.higher_order.executorch_call_delegate"]) + .check(["executorch_exir_dialects_edge__ops_aten_topk_default"]) + .to_executorch() + ) diff --git a/backends/samsung/test/models/test_yolo26_primitives.py b/backends/samsung/test/models/test_yolo26_primitives.py new file mode 100644 index 00000000000..0d730b3ca49 --- /dev/null +++ b/backends/samsung/test/models/test_yolo26_primitives.py @@ -0,0 +1,74 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +import unittest + +import torch +from executorch.backends.samsung.serialization.compile_options import ( + gen_samsung_backend_compile_spec, +) +from executorch.backends.samsung.test.tester import SamsungTester +from executorch.backends.samsung.test.utils.utils import TestConfig + + +class Primitive(torch.nn.Module): + def __init__(self, operator): + super().__init__() + self.operator = operator + + def forward(self, *inputs): + return self.operator(*inputs) + + +class StrideMul(torch.nn.Module): + def __init__(self): + super().__init__() + self.register_buffer("strides", torch.ones(1, 2100)) + + def forward(self, x): + return x * self.strides + + +class TestYolo26Primitives(unittest.TestCase): + def test_compile_primitives(self): + cases = ( + ("attention_softmax", lambda x: torch.softmax(x, -1), ((1, 2, 100, 100),)), + ("query_key_bmm", torch.bmm, ((2, 100, 32), (2, 32, 100))), + ("value_attention_bmm", torch.bmm, ((2, 64, 100), (2, 100, 100))), + ("scores_amax", lambda x: torch.amax(x, dim=-1), ((1, 2100, 80),)), + ( + "indices_topk", + lambda x: torch.topk(x, 300, dim=-1).indices, + ((1, 2100),), + ), + ( + "values_topk", + lambda x: torch.topk(x, 300, dim=-1).values, + ((1, 2100),), + ), + ("both_topk", lambda x: tuple(torch.topk(x, 300)), ((1, 2100),)), + ("stride_mul", StrideMul(), ((1, 4, 2100),)), + ) + for name, operator, shapes in cases: + with self.subTest(operator=name): + inputs = tuple(torch.randn(shape) for shape in shapes) + tester = SamsungTester( + Primitive(operator), + inputs, + [gen_samsung_backend_compile_spec(TestConfig.chipset)], + ) + tester.export().to_edge_transform_and_lower() + if name.endswith("topk"): + tester.check_count( + {"executorch_exir_dialects_edge__ops_aten_topk_default": 1} + ).check_not(["torch.ops.higher_order.executorch_call_delegate"]) + else: + tester.check(["torch.ops.higher_order.executorch_call_delegate"]) + tester.to_executorch() + + +if __name__ == "__main__": + unittest.main() diff --git a/backends/samsung/test/ops/test_mul.py b/backends/samsung/test/ops/test_mul.py index 3db6003b482..81c92152b1a 100644 --- a/backends/samsung/test/ops/test_mul.py +++ b/backends/samsung/test/ops/test_mul.py @@ -76,6 +76,10 @@ def test_fp32_const_mul(self): inputs = (torch.randn(1, 3, 8, 8),) self._test(MulConstant(torch.randn(1, 3, 8, 8)), inputs) + def test_fp32_attention_scale(self): + inputs = (torch.randn(1, 2, 100, 100),) + self._test(MulConstant(32**-0.5), inputs) + def test_fp32_mul_broadcast(self): inputs = (torch.randn(1, 1, 8, 8), torch.randn(1, 3, 8, 8)) self._test(Mul(), inputs) diff --git a/backends/samsung/test/ops/test_topk.py b/backends/samsung/test/ops/test_topk.py index 900bc4cdd89..765fb076d0e 100644 --- a/backends/samsung/test/ops/test_topk.py +++ b/backends/samsung/test/ops/test_topk.py @@ -17,16 +17,81 @@ class TopK(torch.nn.Module): - def __init__(self, k=1, dim=-1) -> None: + def __init__(self, k=1, dim=-1, output="both") -> None: super().__init__() self.k = k self.dim = dim + self.output = output def forward(self, x: torch.Tensor) -> torch.Tensor: - return tuple(torch.topk(x, self.k, dim=self.dim)) + result = torch.topk(x, self.k, dim=self.dim) + if self.output == "indices": + return result.indices + if self.output == "values": + return result.values + return tuple(result) + + +class QuantizedTopK(TopK): + def __init__(self, k=1, dim=-1, output="both", quantize_values=False): + super().__init__(k, dim, output) + self.quantize_values = quantize_values + + @staticmethod + def _qdq(x): + quantized = torch.ops.quantized_decomposed.quantize_per_tensor.default( + x, 0.1, 0, -128, 127, torch.int8 + ) + return torch.ops.quantized_decomposed.dequantize_per_tensor.default( + quantized, 0.1, 0, -128, 127, torch.int8 + ) + + def forward(self, x): + result = super().forward(self._qdq(torch.relu(self._qdq(x)))) + if not self.quantize_values or self.output == "indices": + return result + if self.output == "values": + return self._qdq(result) + return self._qdq(result[0]), result[1] class TestTopK(unittest.TestCase): + def test_a8w8_topk_indices_only(self): + inputs = (torch.randperm(127).float().view(1, 127) * 0.1,) + tester = SamsungTester( + QuantizedTopK(k=30, output="indices"), + inputs, + [gen_samsung_backend_compile_spec(TestConfig.chipset)], + ) + ( + tester.export() + .to_edge_transform_and_lower() + .check_count({"executorch_exir_dialects_edge__ops_aten_topk_default": 1}) + .check_count({"torch.ops.higher_order.executorch_call_delegate": 1}) + .to_executorch() + .run_method_and_compare_outputs(inputs=inputs) + ) + + def test_a8w8_topk_quantized_values(self): + inputs = (torch.randperm(127).float().view(1, 127) * 0.1,) + for output in ("values", "both"): + with self.subTest(output=output): + tester = SamsungTester( + QuantizedTopK(k=30, output=output, quantize_values=True), + inputs, + [gen_samsung_backend_compile_spec(TestConfig.chipset)], + ) + ( + tester.export() + .to_edge_transform_and_lower() + .check_count( + {"executorch_exir_dialects_edge__ops_aten_topk_default": 1} + ) + .check_count({"torch.ops.higher_order.executorch_call_delegate": 1}) + .to_executorch() + .run_method_and_compare_outputs(inputs=inputs) + ) + def _test(self, module: torch.nn.Module, inputs): tester = SamsungTester( module, @@ -37,8 +102,8 @@ def _test(self, module: torch.nn.Module, inputs): tester.export() .check_count({"torch.ops.aten.topk.default": 1}) .to_edge_transform_and_lower() - .check_not(["executorch_exir_dialects_edge__ops_aten_topk_default"]) - .check_count({"torch.ops.higher_order.executorch_call_delegate": 1}) + .check_count({"executorch_exir_dialects_edge__ops_aten_topk_default": 1}) + .check_not(["torch.ops.higher_order.executorch_call_delegate"]) .to_executorch() .run_method_and_compare_outputs(inputs=inputs) ) @@ -54,3 +119,11 @@ def test_fp32_topk_dim_negative1(self): x = torch.randperm(16 * 8 * 8, dtype=torch.float32).view(1, 16, 8, 8) inputs = (x,) self._test(TopK(k=5, dim=-1), inputs) + + def test_fp32_topk_indices_only(self): + inputs = (-torch.randperm(2100, dtype=torch.float32).view(1, 2100),) + self._test(TopK(k=300, output="indices"), inputs) + + def test_fp32_topk_values_only(self): + inputs = (-torch.randperm(2100, dtype=torch.float32).view(1, 2100),) + self._test(TopK(k=300, output="values"), inputs) diff --git a/backends/samsung/test/unit/test_mobilebert_finetune.py b/backends/samsung/test/unit/test_mobilebert_finetune.py new file mode 100644 index 00000000000..00342a6e526 --- /dev/null +++ b/backends/samsung/test/unit/test_mobilebert_finetune.py @@ -0,0 +1,174 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +import tempfile +import unittest +from inspect import signature +from unittest.mock import Mock, patch + +import torch +from datasets import Dataset, DatasetDict +from executorch.backends.samsung.test.models import ( + test_mobilebert_finetuning, + test_mobilebert_qat, +) +from executorch.examples.samsung.scripts import ( + mobilebert_finetune, + mobilebert_finetune_QAT, +) + + +class TinyClassifier(torch.nn.Module): + def __init__(self): + super().__init__() + self.weight = torch.nn.Parameter(torch.ones(())) + self.steps = 0 + self.save_pretrained = Mock() + + def forward(self, texts, attention_mask, labels): + self.steps += 1 + return ((self.weight * texts.float().mean()).square(),) + + +class TestMobileBertFinetune(unittest.TestCase): + def test_qat_pretraining_accepts_custom_learning_rate(self): + finetune = mobilebert_finetune_QAT.MobileBertFinetune.__new__( + mobilebert_finetune_QAT.MobileBertFinetune + ) + datasets = DatasetDict( + train=Dataset.from_dict({"label": [0, 1]}), + validation=Dataset.from_dict({"label": [0, 1]}), + ) + with patch.object( + mobilebert_finetune_QAT, "TrainingArguments" + ) as training_args, patch.object(mobilebert_finetune_QAT, "Trainer"): + finetune.training( + TinyClassifier(), datasets, None, None, learning_rate=5e-4 + ) + self.assertEqual(training_args.call_args.kwargs["learning_rate"], 5e-4) + self.assertEqual( + signature( + mobilebert_finetune_QAT.MobileBertFinetune.get_finetune_mobilebert + ) + .parameters["learning_rate"] + .default, + 2e-5, + ) + + def test_qat_initialization_uses_trainer_seed_and_restores_rng(self): + with torch.random.fork_rng(devices=[]): + torch.manual_seed(43) + state = torch.get_rng_state() + test = test_mobilebert_qat.TestMilestoneMobileBertQAT() + test.setUp() + try: + self.assertEqual(torch.initial_seed(), 42) + finally: + test.tearDown() + self.assertTrue(torch.equal(state, torch.get_rng_state())) + + def test_custom_learning_rate_controls_training_updates(self): + self.assertIn( + "learning_rate", + signature( + mobilebert_finetune.MobileBertFinetune.get_finetune_mobilebert + ).parameters, + ) + finetune = mobilebert_finetune.MobileBertFinetune.__new__( + mobilebert_finetune.MobileBertFinetune + ) + finetune.tokenizer = lambda texts, **kwargs: { + "input_ids": torch.ones(len(texts), 2, dtype=torch.long), + "attention_mask": torch.ones(len(texts), 2, dtype=torch.long), + } + with tempfile.TemporaryDirectory() as directory, patch( + "requests.get", return_value=Mock(content=b"text\t0\nother\t1\n" * 10) + ), patch.object( + mobilebert_finetune.MobileBertForSequenceClassification, + "from_pretrained", + return_value=TinyClassifier(), + ): + model, _ = finetune.get_finetune_mobilebert( + directory, batch_size=4, max_train_samples=8, learning_rate=0.002 + ) + self.assertEqual(model.steps, 10) + self.assertLess(model.weight.item(), 0.99) + + def test_ci_finetuning_does_not_change_global_rng(self): + with torch.random.fork_rng(devices=[]): + torch.manual_seed(43) + state = torch.get_rng_state() + with patch.object( + test_mobilebert_finetuning, "MobileBertFinetune" + ) as finetune, patch.object( + test_mobilebert_finetuning, "SamsungTester" + ), patch.object( + test_mobilebert_finetuning, "gen_samsung_backend_compile_spec" + ): + finetune.return_value.get_finetune_mobilebert.side_effect = ( + lambda *args, **kwargs: (torch.rand(1), None) + ) + test_mobilebert_finetuning.Test_Milestone_MobileBertFinetune().test_mobilebert_finetuning_fp16() + self.assertTrue(torch.equal(state, torch.get_rng_state())) + + def test_limits_training_samples_without_changing_epochs_or_validation(self): + finetune = mobilebert_finetune.MobileBertFinetune.__new__( + mobilebert_finetune.MobileBertFinetune + ) + finetune.tokenizer = lambda texts, **kwargs: { + "input_ids": torch.ones(len(texts), 2, dtype=torch.long), + "attention_mask": torch.ones(len(texts), 2, dtype=torch.long), + } + for limit, expected_steps in ((None, 25), (8, 10), (100, 25)): + with self.subTest(limit=limit), tempfile.TemporaryDirectory() as directory: + model = TinyClassifier() + with patch( + "requests.get", + return_value=Mock(content=b"text\t0\nother\t1\n" * 10), + ), patch.object( + mobilebert_finetune.MobileBertForSequenceClassification, + "from_pretrained", + return_value=model, + ): + _, validation = finetune.get_finetune_mobilebert( + directory, batch_size=4, max_train_samples=limit + ) + self.assertEqual(model.steps, expected_steps) + self.assertEqual(len(validation.dataset), 20) + model.save_pretrained.assert_called_once_with(directory) + + def test_qat_limits_only_pretraining_not_calibration_or_validation(self): + finetune = mobilebert_finetune_QAT.MobileBertFinetune.__new__( + mobilebert_finetune_QAT.MobileBertFinetune + ) + datasets = DatasetDict( + train=Dataset.from_dict({"label": list(range(20))}), + validation=Dataset.from_dict({"label": list(range(10))}), + ) + finetune.load_CSV_dataset = Mock(return_value=(datasets, {"a": 0, "b": 1})) + finetune.tokenizer = None + finetune.metric = None + finetune.batch_size_training = 2 + finetune.num_epochs = 1 + finetune.training = Mock() + with tempfile.TemporaryDirectory() as directory, patch.object( + mobilebert_finetune_QAT.MobileBertForSequenceClassification, + "from_pretrained", + return_value=TinyClassifier(), + ): + _, returned = finetune.get_finetune_mobilebert( + directory, max_train_samples=8, learning_rate=5e-4 + ) + training_data = finetune.training.call_args.args[1] + self.assertEqual(len(training_data["train"]), 8) + self.assertEqual(len(training_data["validation"]), 10) + self.assertEqual(len(returned["train"]), 20) + self.assertEqual(finetune.training.call_args.kwargs["learning_rate"], 5e-4) + finetune.training.return_value.train.assert_called_once() + + +if __name__ == "__main__": + unittest.main() diff --git a/backends/samsung/test/unit/test_mul.py b/backends/samsung/test/unit/test_mul.py new file mode 100644 index 00000000000..de9ba8b5837 --- /dev/null +++ b/backends/samsung/test/unit/test_mul.py @@ -0,0 +1,61 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +import unittest +from unittest.mock import Mock + +import numpy as np +import torch +from executorch.backends.samsung.builders.op_mul import MulVisitor +from executorch.exir import EdgeCompileConfig, to_edge +from executorch.exir.dialects._ops import ops as exir_ops +from executorch.exir.program._fake_program import get_fake_program + + +class Scale(torch.nn.Module): + def forward(self, x): + return x * (32**-0.5) + + +class TestMulVisitor(unittest.TestCase): + def test_partition_support_check_does_not_materialize_fake_constants(self): + program = to_edge( + torch.export.export(Scale(), (torch.randn(1, 2, 100, 100),)), + compile_config=EdgeCompileConfig(_skip_dim_order=True), + ).exported_program() + program = get_fake_program(program) + mul = next( + node + for node in program.graph.nodes + if node.target == exir_ops.edge.aten.mul.Tensor + ) + graph = Mock() + graph.define_tensor.side_effect = range(3) + self.assertTrue(MulVisitor(program).define_node(mul, graph, {})) + self.assertIsNone(graph.define_tensor.call_args_list[1].args[4]) + + def test_attention_scale_has_matching_operand_shapes(self): + shape = (1, 2, 100, 100) + program = to_edge( + torch.export.export(Scale(), (torch.randn(shape),)), + compile_config=EdgeCompileConfig(_skip_dim_order=True), + ).exported_program() + mul = next( + node + for node in program.graph.nodes + if node.target == exir_ops.edge.aten.mul.Tensor + ) + graph = Mock() + graph.define_tensor.side_effect = range(3) + self.assertTrue(MulVisitor(program).define_node(mul, graph, {})) + constant = graph.define_tensor.call_args_list[1] + self.assertEqual(constant.args[1], list(shape)) + np.testing.assert_allclose(constant.args[4], np.full(shape, 32**-0.5)) + self.assertEqual(graph.define_op.call_args.args[2], [0, 1]) + + +if __name__ == "__main__": + unittest.main() diff --git a/backends/samsung/test/unit/test_runtime_executor.py b/backends/samsung/test/unit/test_runtime_executor.py new file mode 100644 index 00000000000..48ae47151d0 --- /dev/null +++ b/backends/samsung/test/unit/test_runtime_executor.py @@ -0,0 +1,134 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +import sys +import tarfile +import tempfile +import time +import unittest +from pathlib import Path +from unittest.mock import patch + +from executorch.backends.samsung.test.utils.runtime_executor import EDBTestManager + + +class TestEDBTestManager(unittest.TestCase): + def test_bundles_inputs_and_uses_four_device_commands(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + for name in ( + "program.pte", + "enn_executor_runner", + "input_0.bin", + "input_1.bin", + ): + (root / name).write_bytes(name.encode()) + with patch( + "executorch.backends.samsung.test.utils.runtime_executor.get_runner_path", + return_value=root / "enn_executor_runner", + ), patch( + "executorch.backends.samsung.test.utils.runtime_executor.subprocess.Popen" + ) as popen: + popen.return_value.communicate.return_value = (b"", b"") + popen.return_value.returncode = 0 + manager = EDBTestManager( + str(root / "program.pte"), + "/data/local/tmp/enn-executorch-test", + ["input_0.bin", "input_1.bin"], + ) + manager.push() + manager.execute() + manager.pull(str(root / "output")) + + self.assertEqual(popen.call_count, 4) + for call in popen.call_args_list: + self.assertTrue(call.kwargs["start_new_session"]) + self.assertEqual( + popen.return_value.communicate.call_args.kwargs, {"timeout": 300} + ) + uploads = [ + call.args[0] for call in popen.call_args_list if "-U" in call.args[0] + ] + self.assertEqual(len(uploads), 1) + with tarfile.open(uploads[0][2]) as bundle: + self.assertEqual( + set(bundle.getnames()), + { + "program.pte", + "enn_executor_runner", + "input_0.bin", + "input_1.bin", + }, + ) + self.assertEqual( + bundle.extractfile("input_1.bin").read(), b"input_1.bin" + ) + command = popen.call_args_list[2].args[0][-1] + self.assertIn("tar -xf", command) + self.assertIn('--input "input_0.bin input_1.bin"', command) + + def test_missing_input_fails_before_device_commands(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "program.pte").write_bytes(b"model") + (root / "enn_executor_runner").write_bytes(b"runner") + with patch( + "executorch.backends.samsung.test.utils.runtime_executor.get_runner_path", + return_value=root / "enn_executor_runner", + ), patch( + "executorch.backends.samsung.test.utils.runtime_executor.subprocess.Popen", + ) as popen: + manager = EDBTestManager( + str(root / "program.pte"), + "/data/local/tmp/enn-executorch-test", + ["missing.bin"], + ) + with self.assertRaises(FileNotFoundError): + manager.push() + popen.assert_not_called() + + @unittest.skipUnless( + sys.platform != "win32", "Devicefarm uses POSIX process groups" + ) + def test_timeout_kills_children_holding_output_pipes(self): + with patch( + "executorch.backends.samsung.test.utils.runtime_executor.get_runner_path", + return_value=Path("enn_executor_runner"), + ), patch.object(EDBTestManager, "COMMAND_TIMEOUT_SECONDS", 0.2, create=True): + manager = EDBTestManager("program.pte", "/data/local/tmp/enn-test", []) + manager.devicefarm = sys.executable + started = time.monotonic() + with self.assertRaisesRegex(RuntimeError, "timed out"): + manager._edb( + [ + "-c", + "import subprocess, sys, time; " + "subprocess.Popen([sys.executable, '-c', " + "'import time; time.sleep(2)']); time.sleep(2)", + ] + ) + self.assertLess(time.monotonic() - started, 1.5) + + def test_device_suites_fail_fast(self): + root = Path(__file__).resolve().parents[4] + script = (root / ".ci/scripts/test-samsung-models.sh").read_text() + self.assertIn("python -m unittest -fv", script) + self.assertIn("python -m unittest discover -fv", script) + + def test_runner_includes_portable_quantization_kernels(self): + root = Path(__file__).resolve().parents[4] + build = (root / "backends/samsung/build.sh").read_text() + runner = (root / "examples/samsung/CMakeLists.txt").read_text() + self.assertIn("-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON", build) + self.assertIn("quantized_ops_lib", runner) + host_build = build.split("function build_android()", 1)[0] + self.assertIn("-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON", host_build) + self.assertIn("-DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=ON", host_build) + self.assertIn("/kernels/quantized/libquantized_ops_aot_lib.so", host_build) + + +if __name__ == "__main__": + unittest.main() diff --git a/backends/samsung/test/unit/test_topk.py b/backends/samsung/test/unit/test_topk.py new file mode 100644 index 00000000000..3f9ac5f44cd --- /dev/null +++ b/backends/samsung/test/unit/test_topk.py @@ -0,0 +1,217 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +import operator +import unittest +from unittest.mock import Mock, patch + +import torch +from executorch.backends.samsung._passes.enn_pass_manager import EnnPassManager +from executorch.backends.samsung.builders.op_topk import TopKVisitor +from executorch.backends.samsung.partition.enn_partitioner import ( + EnnOperatorSupport, + EnnPartitioner, +) +from executorch.backends.samsung.serialization.compile_options import ( + gen_samsung_backend_compile_spec, +) +from executorch.backends.samsung.test.ops.test_topk import QuantizedTopK +from executorch.backends.samsung.utils.constants import QuantConstants +from executorch.exir import EdgeCompileConfig, to_edge +from executorch.exir.dialects._ops import ops as exir_ops + + +class TopK(torch.nn.Module): + def __init__(self, output): + super().__init__() + self.output = output + + def forward(self, x): + result = torch.topk(x, 300) + if self.output == "indices": + return result.indices + if self.output == "values": + return result.values + return result + + +class TestTopKVisitor(unittest.TestCase): + def _is_supported(self, program, node): + with patch( + "executorch.backends.samsung.partition.enn_partitioner.PyEnnWrapper.EnnWrapper", + create=True, + ), patch("executorch.backends.samsung.partition.enn_partitioner.EnnGraph"): + support = EnnOperatorSupport( + program, [gen_samsung_backend_compile_spec("E9955")] + ) + return support.is_node_supported(None, node) + + def test_portable_quantized_topk_serializes(self): + for output in ("indices", "values", "both"): + with self.subTest(output=output): + program = to_edge( + torch.export.export( + QuantizedTopK(k=30, output=output, quantize_values=True), + (torch.randperm(127).float().view(1, 127) * 0.1,), + ), + compile_config=EdgeCompileConfig(_skip_dim_order=True), + ) + try: + serialized = program.to_executorch().buffer + except RuntimeError as error: + self.fail(str(error)) + self.assertGreater(len(serialized), 0) + + def test_quantized_input_without_quantized_values_stays_on_cpu(self): + for output in ("indices", "values", "both"): + with self.subTest(output=output): + program = to_edge( + torch.export.export( + QuantizedTopK(k=300, output=output), (torch.randn(1, 2100),) + ), + compile_config=EdgeCompileConfig(_skip_dim_order=True), + ).exported_program() + topk = next( + node + for node in program.graph.nodes + if node.target == exir_ops.edge.aten.topk.default + ) + self.assertNotIn("quantize_attrs", topk.meta) + self.assertFalse(self._is_supported(program, topk)) + graph_module = EnnPassManager().transform_for_preprocess_pass(program) + topk = next( + node + for node in graph_module.graph.nodes + if node.target == exir_ops.edge.aten.topk.default + ) + self.assertTrue(topk.args[0].meta.get("quantize_attrs")) + self.assertFalse(self._is_supported(program, topk)) + + def test_quantized_values_stay_on_cpu_before_and_after_preprocess(self): + for output in ("values", "both"): + with self.subTest(output=output): + program = to_edge( + torch.export.export( + QuantizedTopK(k=300, output=output, quantize_values=True), + (torch.randn(1, 2100),), + ), + compile_config=EdgeCompileConfig(_skip_dim_order=True), + ).exported_program() + topk = next( + node + for node in program.graph.nodes + if node.target == exir_ops.edge.aten.topk.default + ) + self.assertNotIn("quantize_attrs", topk.meta) + self.assertFalse(self._is_supported(program, topk)) + graph_module = EnnPassManager().transform_for_preprocess_pass(program) + topk = next( + node + for node in graph_module.graph.nodes + if node.target == exir_ops.edge.aten.topk.default + ) + self.assertFalse(self._is_supported(program, topk)) + + def test_output_qdq_stays_with_portable_topk(self): + program = to_edge( + torch.export.export( + QuantizedTopK(k=300, quantize_values=True), (torch.randn(1, 2100),) + ), + compile_config=EdgeCompileConfig(_skip_dim_order=True), + ).exported_program() + topk = next( + node + for node in program.graph.nodes + if node.target == exir_ops.edge.aten.topk.default + ) + values = next(node for node in topk.users if node.args[1] == 0) + quantize = next(iter(values.users)) + dequantize = next(iter(quantize.users)) + for node in (topk, values, quantize, dequantize): + with self.subTest(target=node.target): + self.assertFalse(self._is_supported(program, node)) + + def test_unquantized_topk_stays_on_cpu(self): + for output in ("indices", "values", "both"): + with self.subTest(output=output): + program = to_edge( + torch.export.export(TopK(output), (torch.randn(1, 2100),)), + compile_config=EdgeCompileConfig(_skip_dim_order=True), + ).exported_program() + topk = next( + node + for node in program.graph.nodes + if node.target == exir_ops.edge.aten.topk.default + ) + self.assertFalse(self._is_supported(program, topk)) + + def test_preserves_quantized_output_modes(self): + for output in ("indices", "values", "both"): + with self.subTest(output=output): + program = to_edge( + torch.export.export(TopK(output), (torch.randn(1, 2100),)), + compile_config=EdgeCompileConfig(_skip_dim_order=True), + ).exported_program() + topk = next( + node + for node in program.graph.nodes + if node.target == exir_ops.edge.aten.topk.default + ) + topk.meta["quantize_attrs"] = { + QuantConstants.QUANT_KEY.quant_dtype: torch.int8 + } + graph = Mock() + graph.define_tensor.side_effect = range(3) + ids = {} + self.assertTrue(TopKVisitor(program).define_node(topk, graph, ids)) + sdk_outputs = graph.define_op.call_args.args[3] + self.assertEqual(len(sdk_outputs), 2 if output == "both" else 1) + self.assertEqual( + graph.define_op.call_args.args[4], + { + "k_dims": 300, + "output": {"indices": "index", "values": "value"}.get( + output, "both" + ), + "axis": 1, + "quant_dtype": "AINT8", + }, + ) + for user, sdk_output in zip(topk.users, sdk_outputs): + self.assertEqual(ids[user], sdk_output) + + def test_getitems_follow_cpu_topk_and_downstream_ops_stay_delegated(self): + class TopKAndAdd(torch.nn.Module): + def forward(self, x): + values, indices = torch.topk(x, 300) + return values + 1, indices + + program = to_edge( + torch.export.export(TopKAndAdd(), (torch.randn(1, 2100),)), + compile_config=EdgeCompileConfig(_skip_dim_order=True), + ).exported_program() + with patch( + "executorch.backends.samsung.partition.enn_partitioner.PyEnnWrapper.EnnWrapper", + create=True, + ), patch("executorch.backends.samsung.partition.enn_partitioner.EnnGraph"): + partitions = EnnPartitioner( + [gen_samsung_backend_compile_spec("E9955")] + ).generate_partitions(program) + delegated = {node for partition in partitions for node in partition.nodes} + self.assertTrue(delegated) + self.assertFalse( + any( + node.target in (exir_ops.edge.aten.topk.default, operator.getitem) + for node in delegated + ) + ) + self.assertTrue( + any(node.target == exir_ops.edge.aten.add.Tensor for node in delegated) + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/backends/samsung/test/utils/runtime_executor.py b/backends/samsung/test/utils/runtime_executor.py index 1117ffa3b84..7beaa64e8e5 100644 --- a/backends/samsung/test/utils/runtime_executor.py +++ b/backends/samsung/test/utils/runtime_executor.py @@ -1,6 +1,9 @@ import logging import os +import shlex +import signal import subprocess +import tarfile import tempfile import time @@ -30,6 +33,8 @@ def get_runner_path() -> Path: class EDBTestManager: + COMMAND_TIMEOUT_SECONDS = 300 + def __init__( self, pte_file, @@ -43,25 +48,38 @@ def __init__( self.output_folder = f"{self.work_directory}/output" self.runner = str(get_runner_path()) self.devicefarm = "devicefarm-cli" + self.bundle_path = self.artifacts_dir / "inputs.tar" def _edb(self, cmd): - cmds = [self.devicefarm] - - cmds.extend(cmd) - command = " ".join(cmds) - - logging.info(f"[EDB] Run: {command}") - - result = subprocess.run( - command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE + command = [self.devicefarm, *cmd] + logging.info(f"[EDB] Run: {shlex.join(command)}") + + process = subprocess.Popen( + command, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + start_new_session=True, ) - - stdout_msg = result.stdout.decode("utf-8").strip() + try: + stdout, stderr = process.communicate(timeout=self.COMMAND_TIMEOUT_SECONDS) + except subprocess.TimeoutExpired as error: + # The packaged CLI spawns children that inherit its output pipes. + try: + os.killpg(process.pid, signal.SIGKILL) + except ProcessLookupError: + pass + process.communicate() + raise RuntimeError( + f"EDB command timed out after {self.COMMAND_TIMEOUT_SECONDS}s: " + f"{shlex.join(command)}" + ) from error + + stdout_msg = stdout.decode("utf-8").strip() if stdout_msg: logging.info(f"[EDB] stdout: {stdout_msg}") - if result.returncode != 0: - stderr_msg = result.stderr.decode("utf-8").strip() + if process.returncode != 0: + stderr_msg = stderr.decode("utf-8").strip() if stderr_msg: logging.error(f"[EDB] stderr: {stderr_msg}") raise RuntimeError("edb(Exynos device bridge) command execute failed") @@ -69,47 +87,44 @@ def _edb(self, cmd): logging.info("[EDB] Command succeeded") def push(self): - self._edb(["-E", f"'rm -rf {self.work_directory}'"]) - self._edb(["-E", f"'mkdir -p {self.work_directory}'"]) - self._edb(["-U", self.pte_file, self.work_directory]) - self._edb(["-U", self.runner, self.work_directory]) - self._edb(["-E", f"'ls -al {self.work_directory}'"]) # temp - + files = [Path(self.pte_file), Path(self.runner)] for input_file in self.input_files: - input_file_path = os.path.join(self.artifacts_dir, input_file) - if Path(input_file).name == input_file and os.path.isfile(input_file_path): - # default search the same level directory with pte - self._edb(["-U", input_file_path, self.work_directory]) - elif os.path.isfile(input_file): - self._edb(["-U", input_file, self.work_directory]) - else: + path = Path(input_file) + if path.name == input_file and (self.artifacts_dir / path).is_file(): + path = self.artifacts_dir / path + if not path.is_file(): raise FileNotFoundError(f"Invalid input file path: {input_file}") + files.append(path) + with tarfile.open(self.bundle_path, "w") as bundle: + for path in files: + bundle.add(path, arcname=path.name, recursive=False) + directory = shlex.quote(self.work_directory) + self._edb(["-E", f"rm -rf {directory} && mkdir -p {directory}"]) + self._edb(["-U", str(self.bundle_path), self.work_directory]) def execute(self): - self._edb(["-E", f"'rm -rf {self.output_folder}'"]) - self._edb(["-E", f"'mkdir -p {self.output_folder}'"]) - self._edb(["-E", f"'chmod 777 {self.output_folder}'"]) - # run the delegation input_files_list = " ".join([os.path.basename(x) for x in self.input_files]) enn_executor_runner_args = " ".join( [ - f"--model {os.path.basename(self.pte_file)}", + f"--model {shlex.quote(os.path.basename(self.pte_file))}", f'--input "{input_files_list}"', - f"--output_path {self.output_folder}", + f"--output_path {shlex.quote(self.output_folder)}", ] ) enn_executor_runner_cmd = " ".join( [ - f"'cd {self.work_directory} &&", + f"cd {shlex.quote(self.work_directory)} &&", + f"tar -xf {shlex.quote(self.bundle_path.name)} &&", + f"mkdir -p {shlex.quote(self.output_folder)} &&", + f"chmod 777 {shlex.quote(self.output_folder)} &&", "chmod +x ./enn_executor_runner &&", - f"./enn_executor_runner {enn_executor_runner_args}'", + f"./enn_executor_runner {enn_executor_runner_args}", ] ) self._edb(["-E", f"{enn_executor_runner_cmd}"]) def pull(self, output_path): - self._edb(["-E", f"'ls -al {self.output_folder}'"]) self._edb(["-D", self.output_folder, output_path]) diff --git a/examples/samsung/CMakeLists.txt b/examples/samsung/CMakeLists.txt index 28a6dcc6660..0f2c49a714b 100644 --- a/examples/samsung/CMakeLists.txt +++ b/examples/samsung/CMakeLists.txt @@ -52,8 +52,14 @@ target_include_directories(enn_executor_runner PRIVATE ${EXECUTORCH_ROOT}/..) target_compile_options(enn_executor_runner PRIVATE ${_common_compile_options}) target_link_libraries( - enn_executor_runner PRIVATE enn_logging enn_backend gflags executorch - extension_data_loader portable_ops_lib + enn_executor_runner + PRIVATE enn_logging + enn_backend + gflags + executorch + extension_data_loader + portable_ops_lib + quantized_ops_lib ) set_target_properties( diff --git a/examples/samsung/scripts/mobilebert_finetune.py b/examples/samsung/scripts/mobilebert_finetune.py index bee2909ffb2..26364497018 100644 --- a/examples/samsung/scripts/mobilebert_finetune.py +++ b/examples/samsung/scripts/mobilebert_finetune.py @@ -117,7 +117,9 @@ def build_loader_from_dataset(self, dataset, batch_size, usage="train"): return data_loader - def get_finetune_mobilebert(self, artifacts_dir, batch_size=64): + def get_finetune_mobilebert( + self, artifacts_dir, batch_size=64, max_train_samples=None, learning_rate=1e-5 + ): # Pretrained bert's output ranges in a large scale. It is challenge for enn backend to support directly. # Please finetune mobilebert on specific tasks, make sure that bert's output and hidden states are friendly # to resource-constraint device. @@ -136,6 +138,12 @@ def get_finetune_mobilebert(self, artifacts_dir, batch_size=64): BytesIO(content), delimiter="\t", header=None, names=["text", "label"] ) labels_set = train_data.label.unique() + if max_train_samples is not None: + if max_train_samples <= 0: + raise ValueError("max_train_samples must be positive") + train_data = train_data.sample( + n=min(max_train_samples, len(train_data)), random_state=51 + ) train_data_loader = self.build_loader_from_dataset( train_data, batch_size=batch_size, usage="train" @@ -173,7 +181,7 @@ def get_finetune_mobilebert(self, artifacts_dir, batch_size=64): num_epochs = 5 num_train_steps = len(train_data_loader) * num_epochs - optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5) + optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate) scheduler = get_linear_schedule_with_warmup( optimizer, num_warmup_steps=0, num_training_steps=num_train_steps ) diff --git a/examples/samsung/scripts/mobilebert_finetune_QAT.py b/examples/samsung/scripts/mobilebert_finetune_QAT.py index 40954900feb..f8eec213d83 100644 --- a/examples/samsung/scripts/mobilebert_finetune_QAT.py +++ b/examples/samsung/scripts/mobilebert_finetune_QAT.py @@ -169,13 +169,14 @@ def training( batch_size=8, num_epochs=3, device="cpu", + learning_rate=2e-5, ): # Training arguments training_args = TrainingArguments( output_dir="./results", eval_strategy="epoch", save_strategy="epoch", - learning_rate=2e-5, + learning_rate=learning_rate, per_device_train_batch_size=batch_size, per_device_eval_batch_size=batch_size, num_train_epochs=num_epochs, @@ -197,7 +198,9 @@ def training( ) return trainer - def get_finetune_mobilebert(self, artifacts_dir): + def get_finetune_mobilebert( + self, artifacts_dir, max_train_samples=None, learning_rate=2e-5 + ): # Pretrained bert's output ranges in a large scale. It is challenge for enn backend to support directly. # Please finetune mobilebert on specific tasks, make sure that bert's output and hidden states are friendly # to resource-constraint device. @@ -228,14 +231,22 @@ def get_finetune_mobilebert(self, artifacts_dir): device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model.to(device) + training_datasets = tokenized_datasets.copy() + if max_train_samples is not None: + if max_train_samples <= 0: + raise ValueError("max_train_samples must be positive") + training_datasets["train"] = tokenized_datasets["train"].select( + range(min(max_train_samples, len(tokenized_datasets["train"]))) + ) trainer = self.training( model, - tokenized_datasets, + training_datasets, self.tokenizer, self.compute_metrics, self.batch_size_training, self.num_epochs, device, + learning_rate=learning_rate, ) # Train the model diff --git a/setup.py b/setup.py index 68aee9fc2d0..fba6cedaeeb 100644 --- a/setup.py +++ b/setup.py @@ -342,6 +342,10 @@ def _top_level_package_dirs() -> FrozenSet[str]: "executorch.backends.mlx.test.run_all_tests", "executorch.backends.mlx.test.test_sample", "executorch.backends.mlx.test.test_slot_recycling", + "executorch.backends.samsung.test.models.test_yolo26", + "executorch.backends.samsung.test.models.test_yolo26_primitives", + "executorch.backends.samsung.test.ops.test_mul", + "executorch.backends.samsung.test.ops.test_topk", "executorch.backends.samsung.test.utils.run_tests", "executorch.backends.test.suite.generate_markdown_summary_json", "executorch.examples.models.muse_glimmer.tests.gen_prompt_golden",