CI: Test sdk deserialization against generated aas-core-testdatagen files

.github/actions/sdk-test-testdatagen/action.yml

@@ -0,0 +1,35 @@
+name: sdk-test-testdatagen
+author: hpoeche
+
+description: Fetches generated example files from the aas-specs-metamodel repository and tries to deserialize them
+
+inputs:
+  AAS_SPECS_RELEASE_TAG:
+    required: true
+    description: A tag on the aas-specs-metamodel repository which defines the version of metamodel to test
+
+runs:
+  using: 'composite'
+  steps:
+    - name: Checkout aas-specs-metamodel repository
+      uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 #v5.1.0
+      with:
+        repository: 'admin-shell-io/aas-specs-metamodel'
+        ref: ${{ inputs.AAS_SPECS_RELEASE_TAG }}
+        path: 'aas-specs-metamodel'
+
+    - name: Run tests on example files
+      shell: bash
+      run: |
+        python etc/sdk-testdatagen/deserialization_tests.py \
+          --examples aas-specs-metamodel/schemas \
+          --output sdk-testdatagen-output \
+          --known-deviations etc/sdk-testdatagen/known_deviations.toml \
+          --summary $GITHUB_STEP_SUMMARY
+
+    - name: Upload test output
+      if: ${{ always() }}
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a #v7.0.1
+      with:
+        path: sdk-testdatagen-output
+        name: sdk_testdatagen_results

.github/workflows/main.yml

@@ -47,3 +47,23 @@ jobs:
       - name: Stop and remove the container
         run: |
           docker stop basyx-python-${{ matrix.profile }} && docker rm basyx-python-${{ matrix.profile }}
+
+  sdk-testdatagen:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 #v5.1.0
+      - uses: ./.github/actions/load-versions
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v7
+        with:
+          python-version: ${{ env.X_PYTHON_MIN_VERSION }}
+          cache: "pip"
+          cache-dependency-path: "**/pyproject.toml"
+      - name: Install Python dependencies
+        working-directory: ./sdk
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install .[dev]
+      - uses: ./.github/actions/sdk-test-testdatagen
+        with:
+          AAS_SPECS_RELEASE_TAG: ${{ env.X_AAS_SPECS_RELEASE_TAG }}

.github/workflows/pr.yml

@@ -14,11 +14,11 @@ concurrency:
 
 jobs:
   helper-scripts-static-analysis:
-    #This job runs static code analysis with ruff on the helper scripts in ./etc/scripts
+    #This job runs static code analysis with ruff on the helper scripts in ./etc
     runs-on: ubuntu-latest
     defaults:
       run:
-        working-directory: ./etc/scripts
+        working-directory: ./etc
     steps:
       - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 #v5.1.0
       - uses: ./.github/actions/load-versions

etc/sdk-testdatagen/Readme.md

@@ -0,0 +1,75 @@
+## Fuzz Testing with `aas-core-testdatagen`
+This folder contains a script to test the implementation of the `basyx-python-sdk` against the example files
+generated by the [`aas-core-works/aas-core-testdatagen`](https://github.com/aas-core-works/aas-core-testdatagen) project. In detail, the test script attempts to deserialize
+each example file, keeping track of all occurring errors and generates a summary, showing error frequencies.
+
+### Prerequisites
+The script requires the example files to exist in the specified path in the same structure as they are shipped in the
+[`aas-specs-metamodel`](https://github.com/admin-shell-io/aas-specs-metamodel) repository. Specifically, if `{examples}` is the path passed to the script the following
+directories are scanned for `.json` and `.xml` files respectively:
+
+```
+{examples}/json/examples/generated
+{examples}/xml/examples/generated
+```
+
+Additionally, the `basyx-python-sdk` must be installed via `pip install sdk/.`.
+
+### Running Tests
+Call the Python script with `python etc/sdk-testdatagen/deserialization_tests.py` with the following options:
+
+| Argument     | Required | Description                                                                                                                                                                                   |
+|--------------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `--examples` | Yes      | Path to the directory containing the example files as described in  [Prerequisites](#prerequisites) (usually this is the `/schemas` directory in a cloned `aas-specs-metamodel` respository.) |
+| `--output`   | No       | Directory path where error reports are written                                                                                                                                                |
+| `--summary`  | No       | File path for generated markdown summary (useful in CI as Github step summary)                                                                                                                |
+| `--known-deviations` | No | File path of the known deviations file (default: `known_deviations.toml` in this directory), see [Known Deviations](#known-deviations)                                                    |
+
+### Test Results
+The script distinguishes 5 different results of a test. For all non-succeeding tests, an output report is written to the
+`--output` directory containing the content of the example file that failed, as well as the resulting stack trace.
+Reports are sorted into subdirectories by format (`json`, `xml`) and by their error message
+(e.g. `Value_is_not_a_valid_XSD_date_string`), and named after the subpath of the example file in the `generated`
+directory (e.g. `Extension_OverValueExamples_Date_fuzzed_05.json.txt`).
+
+ - `SUCCESS`: The deserialization threw no exception.
+ - `FAILED`: During deserialization a `KeyError`, `ValueError`, `TypeError` or `AASConstraintViolation` occurred.
+    Indicates that the implementation is not accepting a schema compliant example.
+ - `NOT_IMPLEMENTED`: During deserialization a `NotImplementedError` was thrown. This indicates a well known deviation
+    of implementation and standard. Output reports are grouped in a `NotImplementedError` subdir.
+ - `UNEXPECTED_ERROR`: During deserialization any other exception occurred. It is caught so that the remaining examples
+    are still tested. Output reports are grouped by exception type in an `UnexpectedError` subdir.
+ - `KNOWN_DEVIATION`: The error group of the test is listed in the known deviations file. This overrides the results
+    `FAILED`, `NOT_IMPLEMENTED` and `UNEXPECTED_ERROR`. Output reports are grouped in a `KnownDeviation` subdir.
+
+The script exits with code `1` if at least one test has the result `FAILED` or `UNEXPECTED_ERROR`, if no example files
+are found for a format, or if a known deviation is stale (see below). Otherwise, it exits with `0`.
+`NOT_IMPLEMENTED` and `KNOWN_DEVIATION` results do not make the script fail.
+
+### Known Deviations
+Error groups that are known deviations of the implementation from the standard are listed in
+`known_deviations.toml`. Each entry corresponds to one row of the error groups table in the summary and is listed
+separately per format:
+
+```toml
+# Python's datetime does not support negative years
+[[XML]]
+group = "Negative dates are not supported: Python stdlib datetime requires year >= 1. Report at ..."
+issue = "https://github.com/eclipse-basyx/basyx-python-sdk/issues/<number>"
+```
+
+ - `group`: The exact message of the error group, as shown in the "Group" column of the summary. Messages that contain
+    the fuzzed value need one entry per value.
+ - `issue`: A link to the issue that describes the deviation in detail. It is required.
+
+The reason for the deviation should be briefly stated in a comment above the entry. Other keys are not allowed, and
+the script exits with code `1` before running any test if the file is invalid.
+
+A known deviation whose group does not occur in any test is *stale*. Stale known deviations are listed in the summary
+and make the script fail, so that the file is updated when a deviation is resolved.
+
+## Usage in GitHub CI
+For convenient usage in the GitHub CI a composite action is provided in `.github/actions/sdk-test-testdatagen/action.yml`.
+This action pulls the version of the `aas-specs-metamodel` specified in the `AAS_SPECS_RELEASE_TAG` input variable and
+runs the deserialization tests. The Markdown summary is output as job summary on GitHub and the output reports are
+uploaded as artifact.