diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-09-mxcli-test-tests-list-found-0-test-s-suites-live-tests.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-09-mxcli-test-tests-list-found-0-test-s-suites-live-tests.json new file mode 100644 index 000000000..121abc190 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-09-mxcli-test-tests-list-found-0-test-s-suites-live-tests.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-09", "symptom": "`mxcli test tests --list` prints `Found 0 test(s)` when the suites live in `tests//*.test.mdl`, while `mxcli test tests/Sales --list` finds them", "cause": "`testrunner.ParseTestDir` listed the directory with `os.ReadDir` and `continue`d on every sub-directory, so only test files directly inside the named folder were ever read", "file": "`cmd/mxcli/testrunner/parser.go` (`ParseTestDir`, `walkTestTree`), `cmd/mxcli/testrunner/runner.go` (`emptySuiteError`)", "insight": "**Discovery is a recursive walk now, and hidden folders below the root are skipped** (`.git`, `.mxcli`; the root itself is always read, so `mxcli test .` works). `emptySuiteError` must use the same walk or its `were not read` hint goes blind to exactly the nested layout the fix enables. `filepath.WalkDir` does not follow a root that is a symlink, so `walkTestTree` resolves it first (`os.ReadDir` used to follow it). Layout test: `parser_dir_test.go`; the `FilesRead` counter and the per-file `FileErrors` isolation are unchanged and the walk keeps going past a bad file. No other discovery rule (node_modules etc.) exists in the test runner, so none was added.", "refs": ["mendixlabs/mxcli#1363"]} diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-15-mxcli-test-list-found-0-tests-exits-0-positive-control-folder.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-15-mxcli-test-list-found-0-tests-exits-0-positive-control-folder.json new file mode 100644 index 000000000..a9158b2e1 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-15-mxcli-test-list-found-0-tests-exits-0-positive-control-folder.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-15", "symptom": "`mxcli test tests/ --list` printed `Found 0 test(s)` and exited 0 over a tree whose only suites sat in a deliberate positive-control folder (`tests/unit/_control`) kept out of the default run, and that read as a green result", "cause": "The control was out of the default run only because discovery (before #1363) never entered sub-folders; once discovery recursed there was no opt-out, and `--list` treats an empty listing as success (a real run with 0 tests already exits 1 with `no tests found`) while saying nothing about why", "file": "`cmd/mxcli/testrunner/parser.go` (`walkTestTree`, `TestSuite.SkippedDirs`), `cmd/mxcli/testrunner/runner.go` (`ListTests`, `emptySuiteError`, `skippedDirsLine`)", "insight": "**Making discovery recursive needs an opt-out in the same change, and an opt-out must be reported.** Folders starting with `_` below the root are skipped, like dot-folders (the Go toolchain convention); the root named on the command line is always read, so `mxcli test tests/unit/_control` runs the control. `_` folders are collected into `SkippedDirs` and printed on the line after `Found N test(s)` by both the run and `--list`; dot-folders are not (`.git`/`.mxcli` noise). `ListTests` takes a second writer (`hints`, stderr from the command) and writes `emptySuiteError`'s explanation there when it found nothing, so stdout stays what scripts parse; its exit status is still 0, which is the contract of a listing, while a run with no tests exits 1. Do not fix a 'silent 0' by making `--list` fail: CI uses it to ask what exists. Tests: `parser_dir_test.go` (the `_control` cases, the root-is-read control, the hint writer).", "refs": ["mendixlabs/mxcli#1363"]} diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-15-mxcli-test-one-unreadable-subfolder-fails-the-whole-run.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-15-mxcli-test-one-unreadable-subfolder-fails-the-whole-run.json new file mode 100644 index 000000000..792a3c63d --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-15-mxcli-test-one-unreadable-subfolder-fails-the-whole-run.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-15", "symptom": "`mxcli test tests` fails with `reading test directory: open .../locked: permission denied` and runs nothing when one subfolder (a locked folder, a root-owned folder in a mounted volume) cannot be read, although `tests/Sales/*.test.mdl` next to it is fine", "cause": "`walkTestTree` returned the `filepath.WalkDir` callback's `err` for every entry, so an error below the root became the whole walk's error and `ParseTestDir` returned it as a hard failure", "file": "`cmd/mxcli/testrunner/parser.go` (`walkTestTree`, `ParseTestDir`), `cmd/mxcli/testrunner/runner.go` (`noTestsParsedError`, `reportFileErrors`, `suiteFileErrorResults`)", "insight": "**Moving from a one-level `ReadDir` to a recursive walk turns every subfolder's error into the whole run's error unless errors below the root are made per-entry.** Only an error on the root is fatal; below it `onErr` appends a `FileError{Dir: true}` and the callback returns nil (WalkDir already skips the unreadable folder), so the run still goes red through `FileErrors` and keeps every readable test. Every consumer of `FileErrors` then needs the folder case worded as `could not be read`, not `failed to parse`. **Tests that run as root (CI, devcontainers) cannot see `chmod 000`, and Windows ignores it**: the real chmod test skips there, so the logic is also pinned by injecting the error through the `walkFunc` seam (`walkTestTreeWith`), which runs everywhere. The same walk now serves `testFilesMTime` (`--watch`), so discovery rules cannot drift between the parser and the change detector.", "refs": ["mendixlabs/mxcli#1363", "mendixlabs/mxcli#1368"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 42f7011b9..75adfd826 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Fixed + +- **`mxcli test tests` finds test files in subfolders** (mendixlabs/mxcli#1363) — `mxcli test tests --list` printed `Found 0 test(s)` when the suites live in `tests//*.test.mdl`, while `mxcli test tests/Sales --list` found them: a directory argument only read the files directly inside it. It now reads `*.test.mdl` and `*.test.md` at any depth, in path order, skipping hidden folders (`.git`, `.mxcli`) and folders starting with `_` (the Go convention, so a positive-control suite can sit in `tests/unit/_control` outside the default run; the folder you name is always read, so `mxcli test tests/unit/_control` runs it). Skipped `_` folders are named after the `Found N test(s)` line of a run and at the end of a `--list`, so they do not disappear silently, and `--list` over a tree with no tests now says why on stderr (it still exits 0; a run with no tests still exits 1). A file that cannot be parsed is still reported on its own and does not stop the others; neither does a subfolder that cannot be read (a locked folder, a root-owned one in a mounted volume), which is reported by name and turns the run red while the tests in the other folders still run. The "no tests found" hint now also names misnamed `.mdl` / `.md` files in subfolders and the `_` folders that were skipped. + ## [0.27.0] - 2026-10-15 Headline: **`exec` is all or nothing: when a statement fails at write time, the whole run is rolled back, so a script never leaves a project half-changed.** Alongside it: a wave of `describe` → `check` → `exec` fixes from round-trip runs over real apps (the Siemens FSM template, the GenAI showcase, Marketplace modules), Mendix 9 projects can take microflows, nanoflows and pluggable widgets again, `check -p` resolves many more of the references `exec` and the build would refuse, the classic `filemanager` widget and role-based `Visible: roles (…)` are in MDL, and the companion draws a module's domain model. The MongoDB driver is upgraded for the October 2026 BSON advisories. diff --git a/cmd/mxcli/cmd_test_run.go b/cmd/mxcli/cmd_test_run.go index 567677d76..9a2e883a9 100644 --- a/cmd/mxcli/cmd_test_run.go +++ b/cmd/mxcli/cmd_test_run.go @@ -103,9 +103,18 @@ Examples: # Run tests from a test file mxcli test tests/microflows.test.mdl -p app.mpr - # Run all tests in a directory + # Run all tests in a directory, subfolders included (tests/Sales/*.test.mdl) mxcli test tests/ -p app.mpr + # Discovery is fully recursive: point at the tests folder, not the project + # root (mxcli test . would search deployment/, node_modules/, javasource/ ...). + # A symlinked subfolder is not followed; an unreadable one is reported and + # the tests in the other folders still run. + + # Folders starting with _ below the one you name are skipped (and listed); + # name one to run it, e.g. a positive-control suite + mxcli test tests/unit/_control -p app.mpr + # Output JUnit XML for CI mxcli test tests/ -p app.mpr --junit results.xml @@ -169,7 +178,7 @@ Examples: // resolveTestPaths here too: listing that cannot find a path execution // finds is a confusing split, and `mxcli test tests/ -p app/App.mpr // --list` hit exactly that (mxcli-formula1 findings #15). - if err := testrunner.ListTests(resolveTestPaths(args, projectPath), os.Stdout); err != nil { + if err := testrunner.ListTests(resolveTestPaths(args, projectPath), os.Stdout, os.Stderr); err != nil { fmt.Fprintf(os.Stderr, "Error: %v\n", err) os.Exit(1) } diff --git a/cmd/mxcli/testrunner/parser.go b/cmd/mxcli/testrunner/parser.go index 7ec625fd4..06673490b 100644 --- a/cmd/mxcli/testrunner/parser.go +++ b/cmd/mxcli/testrunner/parser.go @@ -8,6 +8,7 @@ import ( "bufio" "errors" "fmt" + "io/fs" "os" "path/filepath" "regexp" @@ -89,7 +90,8 @@ func (tc TestCase) AssertionCount() int { type TestSuite struct { Name string // Suite name (derived from file name) Tests []TestCase // Test cases - // FileErrors holds one entry per test file that could not be parsed. + // FileErrors holds one entry per test file that could not be parsed, and per + // folder below the root that could not be read (Dir set). // // A malformed file used to abort the whole run, so an unrelated bad file // stopped every other test in the directory from being listed or executed @@ -103,12 +105,24 @@ type TestSuite struct { // made `mxcli test ` say "0 test(s) in 1 file(s)" for a directory nothing // had been read from (ako/mxcli-maintenance §5). FilesRead int + // SkippedDirs lists the folders discovery left out because their name starts + // with "_" (sorted, without duplicates, as the path was reached from the + // command line). A suite that is kept out of the default run on purpose, such + // as a positive control, must not vanish from the output: "Found 0 test(s)" + // over a tree whose tests all sit in tests/unit/_control read as a green run + // (mendixlabs/mxcli#1363). Hidden folders are not listed; they are .git and + // .mxcli noise. + SkippedDirs []string } -// FileError is a test file that could not be parsed, and why. +// FileError is a test file that could not be parsed, or a folder that could not +// be read, and why. type FileError struct { Path string Err error + // Dir is true when Path is a folder that could not be read, rather than a + // test file that could not be parsed. + Dir bool } // ParseTestFile parses a test file (.test.mdl or .test.md) and extracts test cases. @@ -150,35 +164,135 @@ func ParseTestFile(path string) (*TestSuite, error) { }, nil } -// ParseTestDir parses all test files in a directory. +// ParseTestDir parses all test files under a directory, at any depth, in +// lexical path order. A suite per module (tests/Sales/*.test.mdl) is the usual +// layout, so `mxcli test tests` must find what `mxcli test tests/Sales` finds +// (mendixlabs/mxcli#1363). Hidden folders (.git, .mxcli, ...) and folders +// starting with "_" are not entered below the root; the latter are reported in +// SkippedDirs. +// +// A folder below the root that cannot be read is reported in FileErrors (Dir +// set) and the walk carries on, so the tests in the other folders still run. An +// error on the root itself is returned: there is nothing to run. func ParseTestDir(dir string) (*TestSuite, error) { - entries, err := os.ReadDir(dir) + suite := &TestSuite{ + Name: filepath.Base(dir), + } + + skipped, err := walkTestTree(dir, func(path string, d fs.DirEntry) { + if !isTestFile(d.Name()) { + return + } + suite.FilesRead++ + sub, err := ParseTestFile(path) + if err != nil { + suite.FileErrors = append(suite.FileErrors, FileError{Path: path, Err: err}) + return + } + suite.Tests = append(suite.Tests, sub.Tests...) + }, func(path string, err error) { + suite.FileErrors = append(suite.FileErrors, FileError{Path: path, Err: err, Dir: true}) + }) if err != nil { return nil, fmt.Errorf("reading test directory: %w", err) } + suite.SkippedDirs = skipped - suite := &TestSuite{ - Name: filepath.Base(dir), - } + return suite, nil +} - for _, e := range entries { - if e.IsDir() { - continue +// walkTestTree calls fn for every non-directory entry under root, in lexical +// order. Below root it does not enter hidden (dot-prefixed) folders, and does +// not enter folders whose name starts with "_" either, which it returns (as +// reached from root, in walk order). That is the Go toolchain's rule for "./..." +// and it gives a suite a way to stay out of the default run: a deliberate +// positive control lives in tests/unit/_control and is run by naming it. +// +// The root itself is always walked, whatever it is called: `mxcli test .` has a +// root named ".", and `mxcli test tests/_control` is how that control is run. A +// root that is a symlink is followed (as os.ReadDir did); symlinked folders below +// it are not, which keeps a link cycle from looping. +// +// An error reading the root fails the walk: there is nothing to run. An error +// below it belongs to that entry alone: it goes to onErr (nil to ignore it) and +// the walk carries on, because one folder that cannot be read (a locked folder, a +// root-owned one in a mounted volume) must not cost the tests in the others. +// +// Every path handed to fn, onErr and the returned list, and the text of the +// errors, is under root as the caller wrote it, not under the symlink's target. +func walkTestTree(root string, fn func(path string, d fs.DirEntry), onErr func(path string, err error)) ([]string, error) { + return walkTestTreeWith(filepath.WalkDir, root, fn, onErr) +} + +// walkFunc is the shape of filepath.WalkDir, so a test can inject a walk error: +// chmod 000 cannot make a folder unreadable on Windows, or for root. +type walkFunc func(root string, fn fs.WalkDirFunc) error + +func walkTestTreeWith(walk walkFunc, root string, fn func(path string, d fs.DirEntry), onErr func(path string, err error)) ([]string, error) { + walkRoot := root + if info, err := os.Lstat(root); err == nil && info.Mode()&os.ModeSymlink != 0 { + if resolved, err := filepath.EvalSymlinks(root); err == nil { + walkRoot = resolved } - name := e.Name() - if isTestFile(name) { - path := filepath.Join(dir, name) - suite.FilesRead++ - sub, err := ParseTestFile(path) - if err != nil { - suite.FileErrors = append(suite.FileErrors, FileError{Path: path, Err: err}) - continue + } + var skipped []string + err := walk(walkRoot, func(path string, d fs.DirEntry, err error) error { + if err != nil { + err = underTyped(root, walkRoot, err) + if path == walkRoot { + return err + } + if onErr != nil { + onErr(underTypedPath(root, walkRoot, path), err) + } + // WalkDir does not descend into a folder it could not read. + return nil + } + if d.IsDir() { + if path == walkRoot { + return nil + } + if strings.HasPrefix(d.Name(), ".") { + return filepath.SkipDir } - suite.Tests = append(suite.Tests, sub.Tests...) + if strings.HasPrefix(d.Name(), "_") { + skipped = append(skipped, underTypedPath(root, walkRoot, path)) + return filepath.SkipDir + } + return nil + } + if path == walkRoot { + return fmt.Errorf("%s is not a directory", root) } + fn(underTypedPath(root, walkRoot, path), d) + return nil + }) + return skipped, err +} + +// underTypedPath returns path, which the walk reached under walkRoot (root with +// its symlink resolved), as the same path under root. +func underTypedPath(root, walkRoot, path string) string { + if walkRoot == root { + return path + } + if rel, err := filepath.Rel(walkRoot, path); err == nil { + return filepath.Join(root, rel) } + return path +} - return suite, nil +// underTyped does the same for the path an I/O error names, so that the message +// shows the folder the user typed and not where a symlink leads. +func underTyped(root, walkRoot string, err error) error { + if walkRoot == root { + return err + } + var pe *fs.PathError + if errors.As(err, &pe) { + return &fs.PathError{Op: pe.Op, Path: underTypedPath(root, walkRoot, pe.Path), Err: pe.Err} + } + return err } // isTestFile returns true if the filename matches a test file pattern. diff --git a/cmd/mxcli/testrunner/parser_dir_test.go b/cmd/mxcli/testrunner/parser_dir_test.go new file mode 100644 index 000000000..f59b62eb3 --- /dev/null +++ b/cmd/mxcli/testrunner/parser_dir_test.go @@ -0,0 +1,263 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// mendixlabs/mxcli#1363: `mxcli test tests --list` reported "Found 0 test(s)" +// when the suites live in tests//*.test.mdl, while `mxcli test +// tests/Sales --list` found them. ParseTestDir skipped every subdirectory. + +const ( + nestedTestA = "/**\n * @test in A\n */\n$x = 1;\n/\n" + nestedTestB = "# Spec\n\n```mdl-test\n/** @test in B sub */\n$y = 2;\n```\n" + nestedTestTop = "/**\n * @test at top\n */\n$z = 3;\n/\n" + nestedHidden = "/**\n * @test hidden\n */\n$h = 4;\n/\n" +) + +func writeNested(t *testing.T, root, rel, content string) { + t.Helper() + path := filepath.Join(root, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir %s: %v", filepath.Dir(path), err) + } + if err := os.WriteFile(path, []byte(content), 0o600); err != nil { + t.Fatalf("write %s: %v", path, err) + } +} + +func TestParseTestDirFindsTestsInSubfolders(t *testing.T) { + dir := filepath.Join(t.TempDir(), "tests") + writeNested(t, dir, "A/a.test.mdl", nestedTestA) + writeNested(t, dir, "B/sub/b.test.md", nestedTestB) + writeNested(t, dir, "top.test.mdl", nestedTestTop) + writeNested(t, dir, ".hidden/x.test.mdl", nestedHidden) + + suite, err := ParseTestDir(dir) + if err != nil { + t.Fatalf("ParseTestDir: %v", err) + } + if suite.Name != "tests" { + t.Errorf("suite name = %q, want %q", suite.Name, "tests") + } + if suite.FilesRead != 3 { + t.Errorf("FilesRead = %d, want 3 (the hidden folder is not read)", suite.FilesRead) + } + var got []string + for _, tc := range suite.Tests { + got = append(got, tc.Name) + } + // Lexical walk order: A, B/sub, then top.test.mdl. + want := "in A,in B sub,at top" + if strings.Join(got, ",") != want { + t.Errorf("tests = %v, want %s", got, want) + } + if len(suite.FileErrors) != 0 { + t.Errorf("unexpected file errors: %v", suite.FileErrors) + } +} + +func TestParseTestDirBadFileInSubfolderDoesNotStopTheWalk(t *testing.T) { + dir := t.TempDir() + writeNested(t, dir, "A/a.test.mdl", nestedTestA) + writeNested(t, dir, "B/bad.test.mdl", badTestB) + writeNested(t, dir, "C/c.test.mdl", goodTestC) + + suite, err := ParseTestDir(dir) + if err != nil { + t.Fatalf("ParseTestDir returned a hard error: %v", err) + } + if suite.FilesRead != 3 { + t.Errorf("FilesRead = %d, want 3", suite.FilesRead) + } + if len(suite.Tests) != 2 || suite.Tests[0].Name != "in A" || suite.Tests[1].Name != "also good" { + t.Errorf("tests = %+v, want [in A, also good]", suite.Tests) + } + if len(suite.FileErrors) != 1 || !strings.Contains(filepath.ToSlash(suite.FileErrors[0].Path), "B/bad.test.mdl") { + t.Fatalf("FileErrors = %+v, want exactly B/bad.test.mdl", suite.FileErrors) + } +} + +func TestParseTestDirHiddenRootIsStillRead(t *testing.T) { + // Only hidden folders BELOW the root are skipped: `mxcli test .` has a root + // named ".", and a project kept under a dot-folder is the user's choice. + dir := filepath.Join(t.TempDir(), ".suite") + writeNested(t, dir, "a.test.mdl", nestedTestA) + suite, err := ParseTestDir(dir) + if err != nil { + t.Fatalf("ParseTestDir: %v", err) + } + if suite.FilesRead != 1 || len(suite.Tests) != 1 { + t.Errorf("FilesRead = %d, tests = %d, want 1 and 1", suite.FilesRead, len(suite.Tests)) + } +} + +func TestEmptySuiteNamesSkippedCandidatesInSubfolders(t *testing.T) { + dir := t.TempDir() + writeNested(t, dir, "Sales/workflow.mdl", "show entities;\n") + writeNested(t, dir, ".git/ignored.mdl", "show entities;\n") + msg := emptySuiteError([]string{dir}).Error() + if !strings.Contains(msg, "workflow.mdl") { + t.Errorf("nested skipped candidate not named:\n%s", msg) + } + if strings.Contains(msg, "ignored.mdl") { + t.Errorf("file in a hidden folder listed:\n%s", msg) + } +} + +// A folder starting with "_" below the root is skipped, like a dot-folder (the +// Go toolchain's convention), so a deliberate positive-control suite can sit +// beside the real ones without joining the default run. Skipping must not be +// silent: the walk reports what it left out. The root itself is always read. +const nestedControl = "/**\n * @test control\n */\n$c = 5;\n/\n" + +func TestParseTestDirSkipsUnderscoreFoldersAndReportsThem(t *testing.T) { + dir := filepath.Join(t.TempDir(), "tests") + writeNested(t, dir, "unit/a.test.mdl", nestedTestA) + writeNested(t, dir, "unit/_control/c.test.mdl", nestedControl) + writeNested(t, dir, "unit/sub/_x/deep/x.test.mdl", nestedControl) + writeNested(t, dir, "_top/t.test.mdl", nestedControl) + writeNested(t, dir, ".hidden/h.test.mdl", nestedHidden) + + suite, err := ParseTestDir(dir) + if err != nil { + t.Fatalf("ParseTestDir: %v", err) + } + if suite.FilesRead != 1 || len(suite.Tests) != 1 || suite.Tests[0].Name != "in A" { + t.Fatalf("FilesRead = %d, tests = %+v, want only \"in A\"", suite.FilesRead, suite.Tests) + } + var got []string + for _, d := range suite.SkippedDirs { + got = append(got, filepath.ToSlash(strings.TrimPrefix(d, dir))) + } + // Sorted; the dot-folder is noise (.git, .mxcli) and is not reported; a + // skipped folder's own subfolders are not walked, so not listed either. + want := "/_top,/unit/_control,/unit/sub/_x" + if strings.Join(got, ",") != want { + t.Errorf("SkippedDirs = %v, want %s", got, want) + } +} + +func TestParseTestDirReadsAnUnderscoreRoot(t *testing.T) { + // `mxcli test tests/_control` runs the control suite: only folders BELOW the + // root are skipped. + dir := filepath.Join(t.TempDir(), "tests", "_control") + writeNested(t, dir, "c.test.mdl", nestedControl) + writeNested(t, dir, "deeper/d.test.mdl", nestedTestA) + writeNested(t, dir, "_inner/i.test.mdl", nestedTestB) + + suite, err := ParseTestDir(dir) + if err != nil { + t.Fatalf("ParseTestDir: %v", err) + } + if suite.FilesRead != 2 || len(suite.Tests) != 2 { + t.Errorf("FilesRead = %d, tests = %d, want 2 and 2 (root read, _inner skipped)", suite.FilesRead, len(suite.Tests)) + } + if len(suite.SkippedDirs) != 1 || filepath.Base(suite.SkippedDirs[0]) != "_inner" { + t.Errorf("SkippedDirs = %v, want just _inner", suite.SkippedDirs) + } +} + +func TestParseTestFilesDeduplicatesAndSortsSkippedDirs(t *testing.T) { + // The same folder named twice on the command line is reported once. + dir := t.TempDir() + writeNested(t, dir, "b/_b/x.test.mdl", nestedControl) + writeNested(t, dir, "a/_a/x.test.mdl", nestedControl) + suite, err := parseTestFiles([]string{dir, dir}) + if err != nil { + t.Fatalf("parseTestFiles: %v", err) + } + if len(suite.SkippedDirs) != 2 || filepath.Base(suite.SkippedDirs[0]) != "_a" || filepath.Base(suite.SkippedDirs[1]) != "_b" { + t.Errorf("SkippedDirs = %v, want [_a _b]", suite.SkippedDirs) + } +} + +func TestSkippedDirsLine(t *testing.T) { + if got := skippedDirsLine(nil); got != "" { + t.Errorf("no skipped folders: line = %q, want empty", got) + } + got := skippedDirsLine([]string{"tests/unit/_control"}) + want := " Skipped folder(s) starting with _: tests/unit/_control (name one explicitly to run it)" + if got != want { + t.Errorf("line = %q, want %q", got, want) + } + many := []string{"a/_1", "a/_2", "a/_3", "a/_4", "a/_5", "a/_6", "a/_7"} + got = skippedDirsLine(many) + if !strings.Contains(got, "a/_5") || strings.Contains(got, "a/_6") || !strings.Contains(got, "and 2 more") { + t.Errorf("seven folders: want five named and \"and 2 more\":\n%s", got) + } + if strings.Contains(got, "\n") { + t.Errorf("line must be one line:\n%s", got) + } +} + +func TestEmptySuiteNamesAFolderThatHoldsTheTests(t *testing.T) { + // The report that started this: the only suites are in a "_" folder, the + // default run finds nothing, and the message must say where they went. + dir := filepath.Join(t.TempDir(), "tests") + writeNested(t, dir, "unit/_control/c.test.mdl", nestedControl) + msg := emptySuiteError([]string{dir}).Error() + if !strings.Contains(msg, "_control") || !strings.Contains(msg, "name one") { + t.Errorf("message does not name the skipped folder and how to run it:\n%s", msg) + } + if strings.Contains(msg, "were not read") { + t.Errorf("claimed misnamed files where there are none:\n%s", msg) + } + // Control: a tree with no "_" folder does not mention them. + plain := emptySuiteError([]string{dirWith(t)}).Error() + if strings.Contains(plain, "starting with _") { + t.Errorf("empty directory mentions skipped folders:\n%s", plain) + } +} + +func TestListTestsWithZeroTestsExplainsOnTheHintWriterAndSucceeds(t *testing.T) { + dir := filepath.Join(t.TempDir(), "tests") + writeNested(t, dir, "unit/_control/c.test.mdl", nestedControl) + + var out, hints strings.Builder + if err := ListTests([]string{dir}, &out, &hints); err != nil { + t.Fatalf("ListTests returned %v, want nil: --list stays exit 0", err) + } + if !strings.Contains(out.String(), "Found 0 test(s):") { + t.Errorf("stdout lost the count line:\n%s", out.String()) + } + if !strings.Contains(out.String(), "Skipped folder(s) starting with _") { + t.Errorf("stdout does not report the skipped folder:\n%s", out.String()) + } + if !strings.Contains(hints.String(), "no tests found") || !strings.Contains(hints.String(), "_control") { + t.Errorf("hint writer missing the explanation:\n%s", hints.String()) + } + if strings.Contains(out.String(), "no tests found") { + t.Errorf("explanation leaked to stdout, which scripts parse:\n%s", out.String()) + } +} + +func TestListTestsWithTestsWritesNoHint(t *testing.T) { + dir := t.TempDir() + writeNested(t, dir, "a.test.mdl", nestedTestA) + var out, hints strings.Builder + if err := ListTests([]string{dir}, &out, &hints); err != nil { + t.Fatalf("ListTests: %v", err) + } + if hints.String() != "" { + t.Errorf("hint written for a non-empty listing:\n%s", hints.String()) + } +} + +func TestListTestsEveryFileUnparseableSaysSo(t *testing.T) { + // Mirrors Run: a directory whose every file is malformed is not an empty one. + dir := t.TempDir() + writeNested(t, dir, "bad.test.mdl", badTestB) + var out, hints strings.Builder + if err := ListTests([]string{dir}, &out, &hints); err == nil { + t.Fatal("ListTests returned nil for an unparseable file, want the existing non-zero exit") + } + if !strings.Contains(hints.String(), "no tests could be parsed") || strings.Contains(hints.String(), "no tests found") { + t.Errorf("hint = %q, want the \"no tests could be parsed\" variant", hints.String()) + } +} diff --git a/cmd/mxcli/testrunner/parser_file_isolation_test.go b/cmd/mxcli/testrunner/parser_file_isolation_test.go index 38b31ef95..c2fa4ad73 100644 --- a/cmd/mxcli/testrunner/parser_file_isolation_test.go +++ b/cmd/mxcli/testrunner/parser_file_isolation_test.go @@ -165,7 +165,7 @@ func TestListTestsReportsFileErrorsAndFails(t *testing.T) { writeBytes(t, dir, "b_bad.test.mdl", badTestB) var out strings.Builder - err := ListTests([]string{dir}, &out) + err := ListTests([]string{dir}, &out, nil) if err == nil { t.Fatal("ListTests returned nil, want an error so the command exits non-zero") } diff --git a/cmd/mxcli/testrunner/parser_unreadable_test.go b/cmd/mxcli/testrunner/parser_unreadable_test.go new file mode 100644 index 000000000..a8c675865 --- /dev/null +++ b/cmd/mxcli/testrunner/parser_unreadable_test.go @@ -0,0 +1,336 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "errors" + "io/fs" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + "time" +) + +// mendixlabs/mxcli#1368 review: with discovery recursive, one folder that cannot +// be read (a locked folder, a root-owned folder in a mounted volume) must not +// fail the whole run. Below the root the error belongs to that entry; only an +// error on the root itself is fatal. + +func dirEntryOf(t *testing.T, path string) fs.DirEntry { + t.Helper() + info, err := os.Stat(path) + if err != nil { + t.Fatalf("stat %s: %v", path, err) + } + return fs.FileInfoToDirEntry(info) +} + +// fakeWalk replays what filepath.WalkDir does for a tree whose "locked" folder +// cannot be read: the callback is called for the folder first, and again with the +// ReadDir error. It makes the logic testable where chmod 000 does not block +// reading (Windows) or does not apply (root). +func fakeWalk(t *testing.T, permErr error) walkFunc { + t.Helper() + return func(root string, fn fs.WalkDirFunc) error { + sales := filepath.Join(root, "Sales") + locked := filepath.Join(root, "locked") + steps := []struct { + path string + err error + }{ + {root, nil}, + {sales, nil}, + {filepath.Join(sales, "a.test.mdl"), nil}, + {locked, nil}, + {locked, permErr}, + {filepath.Join(root, "z.test.mdl"), nil}, + } + for _, s := range steps { + err := fn(s.path, dirEntryOf(t, s.path), s.err) + if errors.Is(err, fs.SkipDir) { + continue + } + if err != nil { + return err + } + } + return nil + } +} + +func lockedFixture(t *testing.T) string { + t.Helper() + root := filepath.Join(t.TempDir(), "tests") + writeNested(t, root, "Sales/a.test.mdl", nestedTestA) + writeNested(t, root, "z.test.mdl", nestedTestTop) + if err := os.MkdirAll(filepath.Join(root, "locked"), 0o755); err != nil { + t.Fatal(err) + } + return root +} + +func TestWalkTestTreeErrorBelowTheRootIsThatEntrysAndTheWalkContinues(t *testing.T) { + root := lockedFixture(t) + permErr := &fs.PathError{Op: "open", Path: filepath.Join(root, "locked"), Err: fs.ErrPermission} + + var files []string + var errPaths []string + var errs []error + _, err := walkTestTreeWith(fakeWalk(t, permErr), root, + func(path string, d fs.DirEntry) { files = append(files, filepath.Base(path)) }, + func(path string, err error) { errPaths = append(errPaths, path); errs = append(errs, err) }) + if err != nil { + t.Fatalf("walk returned %v, want nil: an error below the root is that entry's", err) + } + if got := strings.Join(files, ","); got != "a.test.mdl,z.test.mdl" { + t.Errorf("files visited = %q, want both readable files", got) + } + if len(errPaths) != 1 || errPaths[0] != filepath.Join(root, "locked") { + t.Fatalf("onErr paths = %v, want just the locked folder", errPaths) + } + if !errors.Is(errs[0], fs.ErrPermission) { + t.Errorf("onErr error = %v, want the permission error", errs[0]) + } +} + +func TestWalkTestTreeErrorBelowTheRootWithoutACallbackIsIgnored(t *testing.T) { + // emptySuiteError walks only to name candidates; it passes no callback. + root := lockedFixture(t) + permErr := &fs.PathError{Op: "open", Path: filepath.Join(root, "locked"), Err: fs.ErrPermission} + var files int + if _, err := walkTestTreeWith(fakeWalk(t, permErr), root, + func(string, fs.DirEntry) { files++ }, nil); err != nil { + t.Fatalf("walk returned %v, want nil", err) + } + if files != 2 { + t.Errorf("files visited = %d, want 2", files) + } +} + +func TestWalkTestTreeErrorOnTheRootIsFatal(t *testing.T) { + root := lockedFixture(t) + rootErr := &fs.PathError{Op: "open", Path: root, Err: fs.ErrPermission} + walk := func(r string, fn fs.WalkDirFunc) error { return fn(r, nil, rootErr) } + called := false + _, err := walkTestTreeWith(walk, root, func(string, fs.DirEntry) {}, + func(string, error) { called = true }) + if !errors.Is(err, fs.ErrPermission) { + t.Fatalf("walk returned %v, want the root's permission error", err) + } + if called { + t.Error("the root's error was handed to onErr; it must fail the run instead") + } +} + +func TestParseTestDirUnreadableSubfolderKeepsTheOtherTests(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("chmod 000 does not stop a directory being read on Windows; the walk logic is covered by TestWalkTestTreeErrorBelowTheRootIsThatEntrysAndTheWalkContinues") + } + if os.Geteuid() == 0 { + t.Skip("running as root: chmod 000 does not stop root reading the folder") + } + root := lockedFixture(t) + locked := filepath.Join(root, "locked") + if err := os.Chmod(locked, 0); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.Chmod(locked, 0o755) }) + + suite, err := ParseTestDir(root) + if err != nil { + t.Fatalf("ParseTestDir returned a hard error: %v", err) + } + if len(suite.Tests) != 2 || suite.FilesRead != 2 { + t.Errorf("tests = %d, FilesRead = %d, want the 2 readable files kept", len(suite.Tests), suite.FilesRead) + } + if len(suite.FileErrors) != 1 || !suite.FileErrors[0].Dir || suite.FileErrors[0].Path != locked { + t.Fatalf("FileErrors = %+v, want exactly the locked folder, marked as a folder", suite.FileErrors) + } + if !errors.Is(suite.FileErrors[0].Err, fs.ErrPermission) { + t.Errorf("error = %v, want a permission error", suite.FileErrors[0].Err) + } + // A folder that cannot be read is not an empty one, and the run stays red. + if res := suiteFileErrorResults(suite.FileErrors); len(res) != 1 || res[0].Status != StatusError { + t.Errorf("results = %+v, want one ERROR so the run is red", res) + } +} + +func TestParseTestDirSymlinkedRootReportsThePathTheUserTyped(t *testing.T) { + target := filepath.Join(t.TempDir(), "realtests") + writeNested(t, target, "A/a.test.mdl", nestedTestA) + writeNested(t, target, "B/bad.test.mdl", badTestB) + writeNested(t, target, "_control/c.test.mdl", nestedControl) + link := filepath.Join(t.TempDir(), "linked") + if err := os.Symlink(target, link); err != nil { + t.Skipf("cannot create a symlink here: %v", err) + } + + suite, err := ParseTestDir(link) + if err != nil { + t.Fatalf("ParseTestDir: %v", err) + } + if len(suite.Tests) != 1 || !strings.HasPrefix(suite.Tests[0].SourceFile, link) { + t.Errorf("tests = %+v, want one test whose SourceFile is under %s", suite.Tests, link) + } + if len(suite.FileErrors) != 1 || !strings.HasPrefix(suite.FileErrors[0].Path, link) { + t.Errorf("FileErrors = %+v, want the bad file under %s", suite.FileErrors, link) + } + if len(suite.SkippedDirs) != 1 || !strings.HasPrefix(suite.SkippedDirs[0], link) { + t.Errorf("SkippedDirs = %v, want the _ folder under %s", suite.SkippedDirs, link) + } + for _, p := range append(suite.SkippedDirs, suite.FileErrors[0].Path, suite.Tests[0].SourceFile) { + if strings.Contains(p, "realtests") { + t.Errorf("path %q shows the resolved target, not the path typed", p) + } + } +} + +func TestWalkTestTreeSymlinkedRootReportsErrorsUnderTheTypedName(t *testing.T) { + target := filepath.Join(t.TempDir(), "realtests") + writeNested(t, target, "Sales/a.test.mdl", nestedTestA) + writeNested(t, target, "z.test.mdl", nestedTestTop) + if err := os.MkdirAll(filepath.Join(target, "locked"), 0o755); err != nil { + t.Fatal(err) + } + link := filepath.Join(t.TempDir(), "linked") + if err := os.Symlink(target, link); err != nil { + t.Skipf("cannot create a symlink here: %v", err) + } + resolved, err := filepath.EvalSymlinks(link) + if err != nil { + t.Fatal(err) + } + permErr := &fs.PathError{Op: "open", Path: filepath.Join(resolved, "locked"), Err: fs.ErrPermission} + + var gotPath string + var gotErr error + _, err = walkTestTreeWith(fakeWalk(t, permErr), link, func(string, fs.DirEntry) {}, + func(path string, err error) { gotPath, gotErr = path, err }) + if err != nil { + t.Fatalf("walk: %v", err) + } + if want := filepath.Join(link, "locked"); gotPath != want { + t.Errorf("onErr path = %q, want %q", gotPath, want) + } + if gotErr == nil || strings.Contains(gotErr.Error(), "realtests") || !strings.Contains(gotErr.Error(), "linked") { + t.Errorf("error text = %v, want the path the user typed, not the resolved one", gotErr) + } + if !errors.Is(gotErr, fs.ErrPermission) { + t.Errorf("rewriting the path lost the cause: %v", gotErr) + } +} + +// The messages say what happened. "file(s) failed to parse" for a folder that +// could not be read sends the reader to a parse error that does not exist. +func TestFailureMessagesDistinguishFoldersFromFiles(t *testing.T) { + dirErr := FileError{Path: "tests/locked", Err: &fs.PathError{Op: "open", Path: "tests/locked", Err: fs.ErrPermission}, Dir: true} + fileErr := FileError{Path: "tests/bad.test.mdl", Err: errors.New("bad block")} + + if got := noTestsParsedError([]FileError{dirErr}).Error(); !strings.Contains(got, "could not be read") || strings.Contains(got, "parse") { + t.Errorf("folders only: %q, want \"could not be read\" and no mention of parsing", got) + } + if got := noTestsParsedError([]FileError{fileErr}).Error(); got != "no tests could be parsed: 1 file(s) failed to parse" { + t.Errorf("files only: %q, want the unchanged message", got) + } + got := noTestsParsedError([]FileError{fileErr, dirErr}).Error() + if !strings.Contains(got, "1 file(s) failed to parse") || !strings.Contains(got, "1 folder(s) could not be read") { + t.Errorf("mixed: %q, want both counts", got) + } + + var out strings.Builder + err := reportFileErrors([]FileError{dirErr}, &out) + if err == nil || strings.Contains(err.Error(), "parsed") || !strings.Contains(err.Error(), "folder(s) could not be read") { + t.Errorf("reportFileErrors error = %v, want a folder wording", err) + } + if strings.Contains(out.String(), "could not be parsed") || !strings.Contains(out.String(), "tests/locked") { + t.Errorf("report:\n%s", out.String()) + } + + res := suiteFileErrorResults([]FileError{dirErr, fileErr}) + if !strings.Contains(res[0].Name, "folder could not be read") || !strings.Contains(res[1].Name, "file could not be parsed") { + t.Errorf("result names = %q, %q", res[0].Name, res[1].Name) + } +} + +// testFilesMTime shares discovery with the parser, so it cannot disagree with it +// about what is a test: hidden and "_" folders below the root are not entered, +// and the root is entered whatever it is called. +func TestTestFilesMTimeFollowsTheDiscoveryRules(t *testing.T) { + old := time.Now().Add(-2 * time.Hour) + now := time.Now() + touch := func(path string, at time.Time) { + t.Helper() + if err := os.Chtimes(path, at, at); err != nil { + t.Fatal(err) + } + } + + dir := filepath.Join(t.TempDir(), "tests") + writeNested(t, dir, "A/a.test.mdl", nestedTestA) + writeNested(t, dir, ".hidden/h.test.mdl", nestedHidden) + writeNested(t, dir, "_control/c.test.mdl", nestedControl) + for _, f := range []string{"A/a.test.mdl", ".hidden/h.test.mdl", "_control/c.test.mdl"} { + touch(filepath.Join(dir, filepath.FromSlash(f)), old) + } + for _, d := range []string{"A", ".hidden", "_control", "."} { + touch(filepath.Join(dir, d), old) + } + before := testFilesMTime([]string{dir}) + + touch(filepath.Join(dir, ".hidden", "h.test.mdl"), now) + touch(filepath.Join(dir, "_control", "c.test.mdl"), now) + for _, d := range []string{"A", ".hidden", "_control", "."} { + touch(filepath.Join(dir, d), old) + } + if after := testFilesMTime([]string{dir}); after.After(before) { + t.Error("an edit in a hidden or _ folder moved the change signal, but discovery never reads those") + } + + touch(filepath.Join(dir, "A", "a.test.mdl"), now) + touch(filepath.Join(dir, "A"), old) + if after := testFilesMTime([]string{dir}); !after.After(before) { + t.Error("an edit in a subfolder produced no change signal") + } + + // The root itself is read whatever it is called, as for the parser. + hiddenRoot := filepath.Join(t.TempDir(), "_suite") + writeNested(t, hiddenRoot, "x.test.mdl", nestedTestA) + touch(filepath.Join(hiddenRoot, "x.test.mdl"), now) + touch(hiddenRoot, old) + if got := testFilesMTime([]string{hiddenRoot}); got.Before(now.Add(-time.Minute)) { + t.Errorf("a root named with a leading _ was not read: mtime = %v", got) + } +} + +// The mapping from the resolved root back to the path the user typed is pure, so +// it is tested without a symlink (creating one needs a privilege on Windows). +func TestUnderTypedMapsAResolvedRootBackToWhatWasTyped(t *testing.T) { + typed := filepath.Join("work", "tests") + resolved := filepath.Join("mnt", "data", "suites") + + if got, want := underTypedPath(typed, resolved, filepath.Join(resolved, "A", "a.test.mdl")), filepath.Join(typed, "A", "a.test.mdl"); got != want { + t.Errorf("underTypedPath = %q, want %q", got, want) + } + if got := underTypedPath(typed, typed, filepath.Join(typed, "A")); got != filepath.Join(typed, "A") { + t.Errorf("no symlink: path changed to %q", got) + } + // A "./tests" root stays as typed when nothing was resolved. + if got := underTypedPath("./tests", "./tests", "tests/A"); got != "tests/A" { + t.Errorf("no symlink: path rewritten to %q", got) + } + + cause := fs.ErrPermission + err := underTyped(typed, resolved, &fs.PathError{Op: "open", Path: filepath.Join(resolved, "locked"), Err: cause}) + if strings.Contains(err.Error(), "suites") || !strings.Contains(err.Error(), filepath.Join(typed, "locked")) { + t.Errorf("error text = %q, want the typed path", err) + } + if !errors.Is(err, cause) { + t.Errorf("the cause was lost: %v", err) + } + plain := errors.New("not a path error") + if got := underTyped(typed, resolved, plain); got != plain { + t.Errorf("an error that names no path was changed to %v", got) + } +} diff --git a/cmd/mxcli/testrunner/runner.go b/cmd/mxcli/testrunner/runner.go index 0ab909a48..0582d2863 100644 --- a/cmd/mxcli/testrunner/runner.go +++ b/cmd/mxcli/testrunner/runner.go @@ -10,6 +10,7 @@ import ( "errors" "fmt" "io" + "io/fs" "os" "os/exec" "path/filepath" @@ -182,6 +183,9 @@ func Run(opts RunOptions) (*SuiteResult, error) { return nil, fmt.Errorf("parsing test files: %w", err) } fmt.Fprintf(w, " Found %d test(s) in %d file(s)\n", len(suite.Tests), suite.FilesRead) + if line := skippedDirsLine(suite.SkippedDirs); line != "" { + fmt.Fprintln(w, line) + } for _, fe := range suite.FileErrors { fmt.Fprintf(w, " ERROR %s: %v\n", fe.Path, fe.Err) } @@ -190,7 +194,7 @@ func Run(opts RunOptions) (*SuiteResult, error) { // A directory whose every file is malformed is not an empty directory, // and must not be reported as one (#903). if len(suite.FileErrors) > 0 { - return nil, fmt.Errorf("no tests could be parsed: %d file(s) failed to parse", len(suite.FileErrors)) + return nil, noTestsParsedError(suite.FileErrors) } return nil, emptySuiteError(opts.TestFiles) } @@ -448,13 +452,27 @@ func writeJUnit(opts RunOptions, result *SuiteResult, w io.Writer) error { } // ListTests parses test files and prints the test names without executing. -func ListTests(files []string, w io.Writer) error { +// +// When nothing was found it also says why, on hints (stderr for the command, so +// a script reading the listing on w is unaffected; nil discards it). The exit +// status does not change: listing an empty tree is not an error, a run of one is +// (#1363). +func ListTests(files []string, w, hints io.Writer) error { suite, err := parseTestFiles(files) if err != nil { return err } fmt.Fprintf(w, "Found %d test(s):\n", len(suite.Tests)) + if len(suite.Tests) == 0 && hints != nil { + // Same decision as Run: a directory whose every file is malformed is not + // an empty one. + if len(suite.FileErrors) > 0 { + fmt.Fprintf(hints, "%v\n", noTestsParsedError(suite.FileErrors)) + } else { + fmt.Fprintf(hints, "%v\n", emptySuiteError(files)) + } + } for _, tc := range suite.Tests { fmt.Fprintf(w, " %s: %s\n", tc.ID, tc.Name) for _, flow := range tc.Setups { @@ -480,21 +498,55 @@ func ListTests(files []string, w io.Writer) error { fmt.Fprintf(w, " (no assertions — this test can only report that the body did not throw)\n") } } + // After the list, so the colon of the Found line still introduces the tests. + if line := skippedDirsLine(suite.SkippedDirs); line != "" { + fmt.Fprintln(w, line) + } return reportFileErrors(suite.FileErrors, w) } -// reportFileErrors prints the files that could not be parsed and returns an -// error when there were any, so listing a partly-readable directory exits -// non-zero instead of looking like a clean listing. +// reportFileErrors prints the files that could not be parsed and the folders +// that could not be read, and returns an error when there were any, so listing a +// partly-readable directory exits non-zero instead of looking like a clean +// listing. func reportFileErrors(errs []FileError, w io.Writer) error { if len(errs) == 0 { return nil } - fmt.Fprintf(w, "\n%d file(s) could not be parsed:\n", len(errs)) + summary := failureSummary(errs) + fmt.Fprintf(w, "\n%s:\n", summary) for _, fe := range errs { fmt.Fprintf(w, " ERROR %s: %v\n", fe.Path, fe.Err) } - return fmt.Errorf("%d test file(s) could not be parsed", len(errs)) + return errors.New(summary) +} + +// failureSummary counts what discovery could not use, in words that fit each +// kind: a test file fails to parse, a folder fails to be read. Calling an +// unreadable folder a parse failure sends the reader looking for a syntax error +// that is not there. +func failureSummary(errs []FileError) string { + files, dirs := countFailures(errs) + var parts []string + if files > 0 { + parts = append(parts, fmt.Sprintf("%d test file(s) could not be parsed", files)) + } + if dirs > 0 { + parts = append(parts, fmt.Sprintf("%d folder(s) could not be read", dirs)) + } + return strings.Join(parts, " and ") +} + +// countFailures splits errs into unparseable test files and unreadable folders. +func countFailures(errs []FileError) (files, dirs int) { + for _, fe := range errs { + if fe.Dir { + dirs++ + } else { + files++ + } + } + return files, dirs } // suiteFileErrorResults turns unparseable files into ERROR results so they are @@ -504,8 +556,12 @@ func reportFileErrors(errs []FileError, w io.Writer) error { func suiteFileErrorResults(errs []FileError) []TestResult { out := make([]TestResult, 0, len(errs)) for _, fe := range errs { + what := "file could not be parsed" + if fe.Dir { + what = "folder could not be read" + } out = append(out, TestResult{ - Name: fmt.Sprintf("%s (file could not be parsed)", filepath.Base(fe.Path)), + Name: fmt.Sprintf("%s (%s)", filepath.Base(fe.Path), what), Status: StatusError, Message: fe.Err.Error(), SourceFile: fe.Path, @@ -537,6 +593,7 @@ func parseTestFiles(paths []string) (*TestSuite, error) { combined.Tests = append(combined.Tests, dirSuite.Tests...) combined.FileErrors = append(combined.FileErrors, dirSuite.FileErrors...) combined.FilesRead += dirSuite.FilesRead + combined.SkippedDirs = append(combined.SkippedDirs, dirSuite.SkippedDirs...) } else { combined.FilesRead++ fileSuite, err := ParseTestFile(path) @@ -554,6 +611,8 @@ func parseTestFiles(paths []string) (*TestSuite, error) { } } + combined.SkippedDirs = sortedUnique(combined.SkippedDirs) + // Re-number test IDs to be globally unique for i := range combined.Tests { combined.Tests[i].ID = fmt.Sprintf("test_%d", i+1) @@ -1016,42 +1075,113 @@ func runCompose(dockerDir string, args ...string) error { // always the intended test under the wrong name, and printing it turns a dead end // into a rename. func emptySuiteError(paths []string) error { - var skipped []string + var skipped, skippedDirs []string for _, path := range paths { info, err := os.Stat(path) if err != nil || !info.IsDir() { continue } - entries, err := os.ReadDir(path) - if err != nil { - continue - } - for _, e := range entries { - if e.IsDir() || isTestFile(e.Name()) { - continue + // Same walk as ParseTestDir, so a misnamed file in tests/Sales/ is + // named too. + dirs, _ := walkTestTree(path, func(file string, d fs.DirEntry) { + if isTestFile(d.Name()) { + return } - if l := strings.ToLower(e.Name()); strings.HasSuffix(l, ".mdl") || strings.HasSuffix(l, ".md") { - skipped = append(skipped, filepath.Join(path, e.Name())) + if l := strings.ToLower(d.Name()); strings.HasSuffix(l, ".mdl") || strings.HasSuffix(l, ".md") { + skipped = append(skipped, file) } - } + }, nil) + skippedDirs = append(skippedDirs, dirs...) } sort.Strings(skipped) + skippedDirs = sortedUnique(skippedDirs) + var msg string if len(skipped) == 0 { - return fmt.Errorf("no tests found: a test file is named *.test.mdl or *.test.md, " + - "and holds one or more `/** @test ... */` blocks") + msg = "no tests found: a test file is named *.test.mdl or *.test.md, " + + "and holds one or more `/** @test ... */` blocks" + } else { + named, more := capNamed(skipped) + msg = fmt.Sprintf("no tests found: discovery matches on the file NAME, so these were not read%s:\n"+ + " %s\n"+ + "Rename one to *.test.mdl (or *.test.md) to have it picked up", + more, strings.Join(named, "\n ")) + } + if len(skippedDirs) > 0 { + // The suites may sit in a folder that was left out on purpose. + named, more := capNamed(slashed(skippedDirs)) + msg += fmt.Sprintf("\nFolders starting with _ are skipped below the folder you name%s; name one to run it:\n %s", + more, strings.Join(named, "\n ")) + } + return errors.New(msg) +} + +// noTestsParsedError is the run's answer when files or folders were found and +// none yielded a test: not an empty directory, so not emptySuiteError. +func noTestsParsedError(errs []FileError) error { + files, dirs := countFailures(errs) + var parts []string + if files > 0 { + parts = append(parts, fmt.Sprintf("%d file(s) failed to parse", files)) + } + if dirs > 0 { + parts = append(parts, fmt.Sprintf("%d folder(s) could not be read", dirs)) + } + if files == 0 { + return fmt.Errorf("no tests could be read: %s", strings.Join(parts, " and ")) } - const maxNamed = 5 - named := skipped + return fmt.Errorf("no tests could be parsed: %s", strings.Join(parts, " and ")) +} + +// maxNamed is how many paths a report names before "and N more". +const maxNamed = 5 + +// capNamed returns at most maxNamed of items and, when some were left out, the +// text " (and N more)" to append to the sentence that lists them. +func capNamed(items []string) (named []string, more string) { + named = items if len(named) > maxNamed { named = named[:maxNamed] } - more := "" - if len(skipped) > len(named) { - more = fmt.Sprintf(" (and %d more)", len(skipped)-len(named)) + if len(items) > len(named) { + more = fmt.Sprintf(" (and %d more)", len(items)-len(named)) + } + return named, more +} + +// skippedDirsLine is the one line that says discovery left folders out. The run +// and --list both print it after "Found N test(s)", so nothing disappears +// silently. Empty when there is nothing to say. +func skippedDirsLine(dirs []string) string { + if len(dirs) == 0 { + return "" } - return fmt.Errorf("no tests found: discovery matches on the file NAME, so these were not read%s:\n"+ - " %s\n"+ - "Rename one to *.test.mdl (or *.test.md) to have it picked up", - more, strings.Join(named, "\n ")) + named, more := capNamed(slashed(dirs)) + return fmt.Sprintf(" Skipped folder(s) starting with _: %s%s (name one explicitly to run it)", + strings.Join(named, ", "), more) +} + +// slashed returns paths with forward slashes, so a line reads the same on every +// platform. +func slashed(paths []string) []string { + out := make([]string, len(paths)) + for i, p := range paths { + out[i] = filepath.ToSlash(p) + } + return out +} + +// sortedUnique sorts items in place and returns the distinct values. +func sortedUnique(items []string) []string { + if len(items) == 0 { + return nil + } + sort.Strings(items) + out := items[:1] + for _, it := range items[1:] { + if it != out[len(out)-1] { + out = append(out, it) + } + } + return out } diff --git a/cmd/mxcli/testrunner/watch.go b/cmd/mxcli/testrunner/watch.go index 84b4e5959..bd7e3e5da 100644 --- a/cmd/mxcli/testrunner/watch.go +++ b/cmd/mxcli/testrunner/watch.go @@ -5,9 +5,9 @@ package testrunner import ( "fmt" "io" + "io/fs" "os" "os/signal" - "path/filepath" "syscall" "time" @@ -169,7 +169,10 @@ func watchMTime(opts RunOptions) time.Time { // // A directory is walked rather than stat'ed: on Linux a directory's own mtime // changes when an entry is created or removed but *not* when an existing file is -// edited in place, which is the common case this loop exists to catch. +// edited in place, which is the common case this loop exists to catch. It is +// walked by walkTestTree, the walk the parser uses, so the two cannot disagree +// about which files are tests (hidden and "_" folders below the root, and the +// root itself, are treated alike). func testFilesMTime(paths []string) time.Time { var newest time.Time for _, p := range paths { @@ -183,15 +186,16 @@ func testFilesMTime(paths []string) time.Time { } continue } - filepath.WalkDir(p, func(path string, d os.DirEntry, err error) error { - if err != nil || d.IsDir() || !isTestFile(d.Name()) { - return nil //nolint:nilerr // an unreadable entry is not a change + // An unreadable entry is not a change: no onErr, and the root's own error + // leaves newest as it was. + _, _ = walkTestTree(p, func(path string, d fs.DirEntry) { + if !isTestFile(d.Name()) { + return } if fi, err := d.Info(); err == nil && fi.ModTime().After(newest) { newest = fi.ModTime() } - return nil - }) + }, nil) // Catch a deletion too: the directory's own mtime moves when an entry // goes away, and the walk above cannot see a file that is no longer there. if info.ModTime().After(newest) { diff --git a/docs-site/src/tools/running-tests.md b/docs-site/src/tools/running-tests.md index 5f37644a6..0a11a1881 100644 --- a/docs-site/src/tools/running-tests.md +++ b/docs-site/src/tools/running-tests.md @@ -82,7 +82,7 @@ first call only stages the change. ## Basic Usage ```bash -# Run all tests in a directory +# Run all tests in a directory, including its subfolders mxcli test tests/ -p app.mpr # Run a specific test file @@ -92,6 +92,42 @@ mxcli test tests/sales.test.mdl -p app.mpr mxcli test tests/integration.test.md -p app.mpr ``` +Discovery walks the directory and every subfolder, in path order. Below the +folder you name it skips folders starting with `.` (`.git`, `.mxcli`) and folders +starting with `_`, the Go convention. Use `_` for a suite that should stay out of +the default run, such as a positive control that is meant to fail: + +```text +tests/unit/Sales/orders.test.mdl # run by `mxcli test tests/` +tests/unit/_control/control.test.mdl # skipped, and named in the output +``` + +```bash +# Run the control on its own: the folder you name is always read +mxcli test tests/unit/_control -p app.mpr +``` + +Skipped `_` folders are named in the output (after the `Found N test(s)` line of +a run, at the end of a `--list`), so they do not disappear silently. A run that +finds no tests exits 1. `--list` exits 0 even when it finds none, and then prints +the reason, including any `_` folders it skipped, on stderr. + +Discovery is fully recursive and has no other skip list, so point `mxcli test` at +the folder that holds your tests (`tests/`), not at the project root. From the +project root, `mxcli test .` searches `deployment/`, `node_modules/`, `javasource/`, `vendorlib/` +and every other folder, which is slow, and runs any stray `*.test.mdl` it finds +there. + +Two more rules follow from the walk: + +- A symlinked folder **below** the folder you name is not followed (the walk + reports it as a file, so it is ignored without a message). A symlinked folder + you name yourself is followed, and errors and listings show the path you typed. +- A folder that cannot be read (a locked folder, a root-owned folder in a mounted + volume) is reported by name as "could not be read", the run exits non-zero, and + the tests in the other folders still run. Only a root that cannot be read stops + the run. + ## Choosing a mode | | Boot cost per run | Database | Needs Docker | diff --git a/docs-site/src/tools/testing.md b/docs-site/src/tools/testing.md index a5ac6dc44..2dbee7dc3 100644 --- a/docs-site/src/tools/testing.md +++ b/docs-site/src/tools/testing.md @@ -26,7 +26,8 @@ Tests execute against a real Mendix runtime. **Docker is one way to get one, not ## Quick Start ```bash -# Run all tests in a directory +# Run all tests in a directory, including its subfolders (folders starting +# with _ are skipped; name one to run it: mxcli test tests/unit/_control) mxcli test tests/ -p app.mpr # Run a specific test file