diff --git a/.claude/skills/mendix/download-marketplace-content/SKILL.md b/.claude/skills/mendix/download-marketplace-content/SKILL.md index 1ef0b887d..8167f097a 100644 --- a/.claude/skills/mendix/download-marketplace-content/SKILL.md +++ b/.claude/skills/mendix/download-marketplace-content/SKILL.md @@ -37,6 +37,7 @@ These are **CLI commands**, not MDL statements. - User wants to upgrade a module that is already installed - User asks whether a marketplace module has been edited locally, or what an upgrade would overwrite - User asks to download a specific `.mpk` (e.g. for CI, or to import in Studio Pro) +- User asks to export a module of their own project as a `.mpk` - User asks which versions of a marketplace item are compatible with their Mendix version - User reports `CE0463` or `CE6087` right after installing or updating a module @@ -443,6 +444,28 @@ Four things this run established, none of them obvious from the command list: The ids are a convenience, not an authority: confirm with `search`/`info` rather than trusting them from memory, and note that the listing name never matches the module name. +## Package a module of the project (`marketplace package`) + +The reverse of `install --file`: export a module as a `.mpk`. No PAT is needed. + +```bash +mxcli marketplace package MyModule -p app.mpr # ./MyModule.mpk +mxcli marketplace package MyModule -p app.mpr -o ./dist --json # {path, sizeBytes, files, foreignEntries} +mxcli marketplace install --file ./dist/MyModule.mpk -p other.mpr # and back in +``` + +- Only the module's own required `userlib/` JARs go in; `--all-libs` bundles every JAR. +- `mx` still exports the whole `resources/` folder (another module's `resources/EventLog/` came + along in a measured export). mxcli warns about each such file in `resources/`, `javasource/` or + `themesource/` and prints the `--exclude` pattern + that drops it, e.g. `--exclude '^resources.EventLog.'`. `--exclude` takes **one** regex (join + with `|`) and the separator must be written `.`: `[/\\]` makes `mx` write nothing. +- The `mx` used must be the project's Mendix version (read from the `mx` itself): a mismatch, or an + `mx` of unknown version, is refused with both versions named. Fix: `mxcli setup mxbuild --version X` + (an `mx` on PATH is used first). `--mxbuild-path` is the override: a mismatch then only warns on stderr. +- `mx` writes into the project it is given, so mxcli runs it on a temporary copy; the project on + disk is unchanged. A Studio Pro lock file beside the project does not stop the export. + ## Notes - `install`/`update`/`diff` require `-p `; `download` does not. diff --git a/CHANGELOG.md b/CHANGELOG.md index 42f7011b9..861a4a42f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Added + +- **`mxcli marketplace package` — export a module as a `.mpk`** (mendixlabs/mxcli#1382) — `mxcli marketplace package MyModule -p app.mpr [-o dir] [--all-libs] [--exclude REGEX] [--exclude-managed-dependencies] [--json]` is the reverse of `marketplace install --file`: it wraps Mendix's own `mx create-module-package` (resolved for the project's Mendix version), so the package is what Studio Pro's *Export module package* makes, and it installs back with `install --file`. By default only the module's own required `userlib/` JARs go in (`mx -l`; measured 12.5 KB against 24 MB with Business Events installed); `--all-libs` bundles every JAR. `-l` filters only JARs — mx still exports the app's whole `resources/` folder (measured: another module's `resources/EventLog/` came along) — so after the export mxcli opens the package, warns on stderr about each `resources//`, `javasource//` or `themesource//` file of another module, and prints the `--exclude` pattern that drops them in the one form mx accepts (`^resources.EventLog.`; a character class or backslash is rejected by mx with exit 1 and a usage message, and an exit 0 that wrote no package is reported as an error too). A module name that is not in the project lists the project's modules instead of failing with mx's exit 3, and mx's exit codes 1–4 are reported with what they mean and what mx printed. `mx create-module-package` writes into the project it is given — measured on a blank 11.12.5 app: `theme-cache/web/theme.compiled.css` and its map rewritten, `deployment/sass/main.scss` added, the module's `themesource/` folder created, model files untouched — so it runs on a temporary copy, as `docker check` does; a before/after listing of the project (sizes and modification times) was identical. The `mx` that runs must be the project's Mendix version: `ResolveMxForVersion` prefers an `mx` on PATH and falls back to any cached one, so mxcli reads the version of the `mx` it found (its `mx.deps.json`, else its version-named folder) and refuses a different major.minor.patch, or one it cannot identify, naming both versions and the fix (`mxcli setup mxbuild --version X`); `--mxbuild-path` is the override, with a warning on stderr. mx's own output on a successful export goes to stderr, cancelling the command stops mx, and an output directory inside the project is not copied into the temporary copy. Round trip on 11.12.5: export, `install --file` into a second fresh app, `mx check` — 0 errors. + ## [0.27.0] - 2026-10-15 Headline: **`exec` is all or nothing: when a statement fails at write time, the whole run is rolled back, so a script never leaves a project half-changed.** Alongside it: a wave of `describe` → `check` → `exec` fixes from round-trip runs over real apps (the Siemens FSM template, the GenAI showcase, Marketplace modules), Mendix 9 projects can take microflows, nanoflows and pluggable widgets again, `check -p` resolves many more of the references `exec` and the build would refuse, the classic `filemanager` widget and role-based `Visible: roles (…)` are in MDL, and the companion draws a module's domain model. The MongoDB driver is upgraded for the October 2026 BSON advisories. diff --git a/cmd/mxcli/cmd_marketplace.go b/cmd/mxcli/cmd_marketplace.go index 3a9cb256c..3a80a2a8a 100644 --- a/cmd/mxcli/cmd_marketplace.go +++ b/cmd/mxcli/cmd_marketplace.go @@ -28,7 +28,9 @@ Requires a Personal Access Token (PAT). Run 'mxcli auth login' first. 'marketplace download' fetches a content version's .mpk to disk; 'marketplace install --file ' installs a .mpk from disk — a downloaded one, or any package distributed as a file — through the same writer as an online -install, with no PAT.`, +install, with no PAT. +'marketplace package ' is the reverse: it exports a module of a project +as a .mpk.`, } var marketplaceSearchCmd = &cobra.Command{ diff --git a/cmd/mxcli/cmd_marketplace_package.go b/cmd/mxcli/cmd_marketplace_package.go new file mode 100644 index 000000000..2b020b2c2 --- /dev/null +++ b/cmd/mxcli/cmd_marketplace_package.go @@ -0,0 +1,522 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "archive/zip" + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "regexp" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + "github.com/spf13/cobra" +) + +var marketplacePackageCmd = &cobra.Command{ + Use: "package -p ", + Short: "Export a module of a project as a .mpk module package", + Long: `Export a module as a .mpk module package, the reverse of +'marketplace install --file '. The package is made by Mendix's own +'mx create-module-package', so Studio Pro, 'mx module-import' and mxcli can all +read it back. + + - Without --all-libs only the userlib JARs the module declares it needs (those + with a '..RequiredLib' file beside them) are bundled; with --all-libs + every JAR in the app's userlib/ goes in (24 MB on an app with Business Events + installed, against 12 KB for the module's own). + - mx still exports the app's whole resources/ folder, whatever --all-libs says + (measured: resources/EventLog/ of another module came along). mxcli opens the + package and warns about any resources/, javasource/ or themesource/ file that + belongs to another module, and prints the --exclude pattern that drops it. + - --exclude takes ONE regular expression, matched against each file's path + inside the package; join alternatives with '|'. Write the folder separator as + '.' ('^resources.EventLog.'): a character class or a backslash makes mx print + its usage and write nothing. + +The mx must be the Mendix version of the project: a package made by another +version than the app it comes from is refused, naming both versions. mx is taken +from, in order, an mx on PATH, a Studio Pro install and the mxbuild cache, so a +different version can be found first; 'mxcli setup mxbuild --version ' +installs the right one. --mxbuild-path is the override: with it a mismatch is +allowed, with a warning on stderr. + +mx writes into the project it is given (theme-cache/, deployment/sass/, the +module's themesource/ folder), so it runs on a temporary copy; the project on +disk is not changed. An output directory inside the project is left out of that +copy. A Studio Pro lock file beside the project does not stop the +export, which reads what is saved on disk. + +The package is named .mpk and is written to --output-dir (default: the +current directory, created if missing), replacing an earlier one.`, + Example: ` mxcli marketplace package MyModule -p app.mpr + mxcli marketplace package MyModule -p app.mpr -o ./dist + mxcli marketplace package MyModule -p app.mpr --exclude '^resources.EventLog.|^themesource.atlas_core.' + mxcli marketplace package MyModule -p app.mpr --all-libs --json`, + Args: cobra.ExactArgs(1), + RunE: runMarketplacePackage, + // Like install: a failed export is a runtime failure, not a misuse of the + // command, and main() already prints the error. + SilenceUsage: true, + SilenceErrors: true, +} + +func init() { + marketplacePackageCmd.Flags().StringP("project", "p", "", "path to the Mendix project (.mpr)") + marketplacePackageCmd.Flags().StringP("output-dir", "o", ".", "directory to write .mpk to (created if missing)") + marketplacePackageCmd.Flags().Bool("all-libs", false, + "bundle every userlib JAR of the app, not only those the module declares it requires") + marketplacePackageCmd.Flags().String("exclude", "", + "regular expression for files to leave out, one only (join alternatives with '|'); write the folder separator as '.'") + marketplacePackageCmd.Flags().Bool("exclude-managed-dependencies", false, + "leave out the module's managed dependencies (mx -m)") + marketplacePackageCmd.Flags().String("mxbuild-path", "", + "path to the Mendix installation to take mx from; it overrides the version check (a mismatch only warns)") + marketplacePackageCmd.Flags().Bool("json", false, "emit {path, sizeBytes, files, foreignEntries} as JSON") + _ = marketplacePackageCmd.MarkFlagRequired("project") + + marketplaceCmd.AddCommand(marketplacePackageCmd) +} + +func runMarketplacePackage(cmd *cobra.Command, args []string) error { + var opts packageOptions + opts.Module = args[0] + opts.ProjectPath, _ = cmd.Flags().GetString("project") + opts.OutputDir, _ = cmd.Flags().GetString("output-dir") + opts.AllLibs, _ = cmd.Flags().GetBool("all-libs") + opts.Exclude, _ = cmd.Flags().GetString("exclude") + opts.ExcludeManaged, _ = cmd.Flags().GetBool("exclude-managed-dependencies") + opts.MxBuildPath, _ = cmd.Flags().GetString("mxbuild-path") + opts.JSON, _ = cmd.Flags().GetBool("json") + _, err := runPackage(cmd.Context(), opts, cmd.OutOrStdout(), cmd.ErrOrStderr()) + return err +} + +type packageOptions struct { + ProjectPath string + Module string + OutputDir string + AllLibs bool + Exclude string + ExcludeManaged bool + MxBuildPath string + JSON bool +} + +// packageResult is what the export produced. ForeignEntries is never nil, so the +// JSON form carries an empty array rather than null. +type packageResult struct { + Path string `json:"path"` + SizeBytes int64 `json:"sizeBytes"` + Files int `json:"files"` + ForeignEntries []string `json:"foreignEntries"` +} + +// The variables below are seams: tests replace them, because they cannot run +// Studio Pro's mx. + +// packageProjectInfo reports the project's Mendix version and its module names. +var packageProjectInfo = func(mprPath string) (version string, modules []string, err error) { + reader, err := openProjectReadOnly(mprPath) + if err != nil { + return "", nil, err + } + defer reader.Disconnect() + version, _ = reader.GetMendixVersion() + mods, err := reader.ListModules() + if err != nil { + return "", nil, err + } + for _, m := range mods { + modules = append(modules, m.Name) + } + return version, modules, nil +} + +var resolveMxForPackage = docker.ResolveMxForVersion + +// mxBinaryVersion reports the Mendix version of an mx binary, "" when unknown. +var mxBinaryVersion = docker.MxBinaryVersion + +// mxCreateModulePackageCmd runs mx with args and writes what it prints (stdout +// and stderr together; mx reports errors on stdout) to out. It returns mx's exit +// code; err is set when mx could not be started or ctx ended before it did (mx +// is then killed). +var mxCreateModulePackageCmd = func(ctx context.Context, mxPath string, args []string, out io.Writer) (exitCode int, err error) { + c := exec.CommandContext(ctx, mxPath, args...) + c.Stdout = out + c.Stderr = out + docker.PrepareMxCommand(c) + runErr := c.Run() + if ctxErr := ctx.Err(); ctxErr != nil { + return -1, ctxErr + } + var exitErr *exec.ExitError + switch { + case runErr == nil: + return 0, nil + case errors.As(runErr, &exitErr): + return exitErr.ExitCode(), nil + } + return -1, runErr +} + +// packageStagingPrefix names the directory inside the output directory that mx +// writes into. Exporting to a fresh directory, then moving the package out, is +// how a package left over from an earlier run is told apart from the one this +// run made, and guards against an mx that exits 0 without writing a package (it +// exits 1 for a pattern it cannot parse, measured on 11.12.5). +const packageStagingPrefix = ".mxcli-package-" + +func runPackage(ctx context.Context, opts packageOptions, stdout, stderr io.Writer) (*packageResult, error) { + if _, err := os.Stat(opts.ProjectPath); err != nil { + return nil, fmt.Errorf("project not found: %s", opts.ProjectPath) + } + projectAbs, err := filepath.Abs(opts.ProjectPath) + if err != nil { + return nil, err + } + progress := stdout + if opts.JSON { + progress = io.Discard // stdout carries one JSON document and nothing else + } + + version, modules, err := packageProjectInfo(opts.ProjectPath) + if err != nil { + return nil, fmt.Errorf("open project: %w", err) + } + module, ok := findModuleName(modules, opts.Module) + if !ok { + sorted := append([]string(nil), modules...) + sort.Strings(sorted) + return nil, fmt.Errorf("module %q not found in %s\nmodules in the project: %s", + opts.Module, filepath.Base(opts.ProjectPath), strings.Join(sorted, ", ")) + } + + if opts.MxBuildPath != "" { + // ResolveMxForVersion quietly falls back to an mx on PATH when this path is + // bad, which would turn a typo into an unchecked export. + if _, err := os.Stat(opts.MxBuildPath); err != nil { + return nil, fmt.Errorf("--mxbuild-path %s: %w", opts.MxBuildPath, err) + } + } + mxPath, err := resolveMxForPackage(opts.MxBuildPath, version) + if err != nil { + return nil, fmt.Errorf("locate mx for Mendix %s: %w\nhint: run 'mxcli setup mxbuild -p %s'", version, err, opts.ProjectPath) + } + mxVersion, err := checkMxVersion(mxPath, version, opts.MxBuildPath != "", stderr) + if err != nil { + return nil, err + } + + outDir, err := filepath.Abs(opts.OutputDir) + if err != nil { + return nil, err + } + if err := os.MkdirAll(outDir, 0o755); err != nil { + return nil, fmt.Errorf("create output directory: %w", err) + } + if err := ctx.Err(); err != nil { + return nil, err + } + + // mx writes into the project it is given, so it runs on a copy; see the + // command's Long text. The staging directory is made after the copy, and an + // output directory inside the project is left out of it, so neither the staging + // directory nor earlier packages are copied. + var finalMpk, mxOutput string + var exitCode int + err = docker.WithProjectCopy(projectAbs, func(workMpr string) error { + staging, err := os.MkdirTemp(outDir, packageStagingPrefix+"*") + if err != nil { + return fmt.Errorf("create staging directory in %s: %w", outDir, err) + } + defer os.RemoveAll(staging) + + fmt.Fprintf(progress, "Exporting module %s from %s (Mendix %s) with %s...\n", + module, filepath.Base(projectAbs), version, mxDescription(mxPath, mxVersion)) + var out bytes.Buffer + exitCode, err = mxCreateModulePackageCmd(ctx, mxPath, mxArgs(opts, staging, workMpr, module), &out) + if err != nil { + return fmt.Errorf("run %s: %w", mxPath, err) + } + mxOutput = strings.TrimSpace(nameUserPaths(out.String(), filepath.Dir(workMpr), filepath.Dir(projectAbs), staging, outDir)) + if exitCode != 0 { + return nil + } + found, _ := filepath.Glob(filepath.Join(staging, "*.mpk")) + if len(found) != 1 { + return fmt.Errorf("mx exited 0 but wrote no .mpk%s", withMxOutput(mxOutput)) + } + dest := filepath.Join(outDir, filepath.Base(found[0])) + if err := os.Rename(found[0], dest); err != nil { + return fmt.Errorf("move the package to %s: %w", dest, err) + } + finalMpk = dest + if mxOutput != "" { + // mx's warnings on a successful export; stderr, so --json stays clean. + fmt.Fprintln(stderr, mxOutput) + } + return nil + }, outDir) + if err != nil { + return nil, err + } + if exitCode != 0 { + return nil, mxExitError(exitCode, module, mxOutput) + } + + names, err := readPackageEntries(finalMpk) + if err != nil { + return nil, fmt.Errorf("read the package %s: %w", finalMpk, err) + } + foreign, own := classifyPackageEntries(names, module) + info, err := os.Stat(finalMpk) + if err != nil { + return nil, err + } + res := &packageResult{Path: finalMpk, SizeBytes: info.Size(), Files: len(names), ForeignEntries: foreign} + + if len(foreign) > 0 { + warnForeignEntries(stderr, module, foreign, own, opts.Exclude) + } + if opts.JSON { + enc := json.NewEncoder(stdout) + enc.SetIndent("", " ") + return res, enc.Encode(res) + } + fmt.Fprintf(stdout, "Exported module %s\n path: %s\n size: %s (%d bytes)\n files: %d\n", + module, res.Path, formatBytes(res.SizeBytes), res.SizeBytes, res.Files) + return res, nil +} + +// mxArgs builds the argument list of 'mx create-module-package': options first, +// then the project and the module as the positional arguments mx expects. +func mxArgs(opts packageOptions, packageDir, targetMpr, module string) []string { + args := []string{"create-module-package"} + if !opts.AllLibs { + args = append(args, "-l") + } + if opts.Exclude != "" { + args = append(args, "-e", opts.Exclude) + } + if opts.ExcludeManaged { + args = append(args, "-m") + } + return append(args, "-d", packageDir, targetMpr, module) +} + +// checkMxVersion compares the Mendix version of the mx that was found with the +// project's and returns the mx's version ("" when unknown). A mismatch, or an mx +// whose version cannot be told, is refused: the package would be made by another +// Mendix version than the app it comes from. override is true when the user named +// the mx with --mxbuild-path; then the same findings are only warned about on +// stderr (which --json leaves alone). +func checkMxVersion(mxPath, projectVersion string, override bool, stderr io.Writer) (string, error) { + got := mxBinaryVersion(mxPath) + if projectVersion == "" { + fmt.Fprintf(stderr, "warning: the project's Mendix version is unknown, so the version of %s was not checked\n", mxPath) + return got, nil + } + if docker.SameMendixRelease(got, projectVersion) { + return got, nil + } + var problem string + if got == "" { + problem = fmt.Sprintf("cannot tell which Mendix version %s is, and the project is Mendix %s", mxPath, projectVersion) + } else { + problem = fmt.Sprintf("the mx found, %s, is Mendix %s, but the project is Mendix %s", mxPath, got, projectVersion) + } + if override { + fmt.Fprintf(stderr, "warning: %s; going on because --mxbuild-path was given\n", problem) + return got, nil + } + return "", fmt.Errorf("%s.\nA package made by another Mendix version than the app it comes from is refused.\n"+ + "hint: run 'mxcli setup mxbuild --version %s' to install the matching toolchain (an mx on PATH is used before it,\n"+ + "so take that one off PATH); to use this mx on purpose, pass --mxbuild-path", problem, projectVersion) +} + +// mxDescription names the mx for the progress line: its path, and its version when known. +func mxDescription(mxPath, mxVersion string) string { + if mxVersion == "" { + return mxPath + } + return fmt.Sprintf("%s (Mendix %s)", mxPath, mxVersion) +} + +// findModuleName matches a module name exactly, then case-insensitively, and +// returns the project's own spelling, which is what mx wants. +func findModuleName(modules []string, name string) (string, bool) { + for _, m := range modules { + if m == name { + return m, true + } + } + for _, m := range modules { + if strings.EqualFold(m, name) { + return m, true + } + } + return "", false +} + +// nameUserPaths rewrites mx's output to name the files the user has: the +// temporary copy becomes the project, the staging directory the output directory. +func nameUserPaths(text, workDir, projectDir, staging, outDir string) string { + return strings.NewReplacer(workDir, projectDir, staging, outDir).Replace(text) +} + +func withMxOutput(output string) string { + if output == "" { + return "" + } + return "\nmx printed:\n" + output +} + +// mxExitError turns the documented exit codes of 'mx create-module-package' +// into messages that say what to change. +func mxExitError(code int, module, output string) error { + var msg string + switch code { + case 1: + msg = "mx could not use the project file (exit 1): it is not a usable Mendix project, or an option was rejected (a --exclude pattern mx cannot parse fails this way)" + case 2: + msg = "mx could not determine the output folder (exit 2)" + case 3: + msg = fmt.Sprintf("mx rejected the module name %q (exit 3)", module) + case 4: + msg = "mx reported an unknown export error (exit 4)" + default: + msg = fmt.Sprintf("mx create-module-package failed (exit status %d)", code) + } + return errors.New(msg + withMxOutput(output)) +} + +// packageModuleRoots are the package folders that hold one sub-folder per +// module. Only resources/ was observed leaking another module's files into a +// package (11.12.5); javasource/ and themesource/ are checked as well, in case an +// mx version exports them. Mendix writes those two folder names in lower case. +var packageModuleRoots = map[string]bool{"resources": true, "javasource": true, "themesource": true} + +// readPackageEntries lists the files in the package, folder entries left out, +// with '/' as the separator. +func readPackageEntries(mpkPath string) ([]string, error) { + zr, err := zip.OpenReader(mpkPath) + if err != nil { + return nil, err + } + defer zr.Close() + var names []string + for _, f := range zr.File { + name := strings.ReplaceAll(f.Name, `\`, "/") + if strings.HasSuffix(name, "/") { + continue + } + names = append(names, name) + } + return names, nil +} + +// classifyPackageEntries splits entries into those under another module's +// folder — resources//…, javasource//…, themesource//… with X not the +// exported module, compared case-insensitively — and everything else. Both are +// sorted; foreign is never nil. +func classifyPackageEntries(names []string, module string) (foreign, own []string) { + foreign = []string{} + for _, name := range names { + parts := strings.Split(name, "/") + if len(parts) >= 3 && packageModuleRoots[parts[0]] && !strings.EqualFold(parts[1], module) { + foreign = append(foreign, name) + } else { + own = append(own, name) + } + } + sort.Strings(foreign) + sort.Strings(own) + return foreign, own +} + +var nonWordChar = regexp.MustCompile(`[^A-Za-z0-9_]`) + +// excludePatternFor builds the one --exclude expression that drops the foreign +// entries, in the form mx accepts everywhere: anchored, with '.' for the folder +// separator and for any character that is not a letter, digit or underscore — +// never a character class or a backslash. Because '.' matches any character, a +// folder whose pattern would also match one of the module's own files is left +// out of the pattern and returned in ambiguous instead. +func excludePatternFor(foreign, own []string) (pattern string, ambiguous []string) { + groups := map[string][]string{} // "resources/EventLog" -> its entries + for _, name := range foreign { + parts := strings.Split(name, "/") + key := parts[0] + "/" + parts[1] + groups[key] = append(groups[key], name) + } + keys := make([]string, 0, len(groups)) + for k := range groups { + keys = append(keys, k) + } + sort.Strings(keys) + + var patterns []string + for _, key := range keys { + p := "^" + nonWordChar.ReplaceAllString(key, ".") + "." + re := regexp.MustCompile(p) + clash := false + for _, o := range own { + if re.MatchString(o) { + clash = true + break + } + } + if clash { + ambiguous = append(ambiguous, groups[key]...) + continue + } + patterns = append(patterns, p) + } + return strings.Join(patterns, "|"), ambiguous +} + +// shellQuote quotes s as one word for a POSIX shell: in single quotes, with each +// single quote inside closing the quote, adding an escaped quote, and reopening it. +func shellQuote(s string) string { + return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'" +} + +// warnForeignEntries tells the user the package holds files of other modules and +// how to leave them out. It is a warning, not an error: the package is valid, +// just larger than the module. +func warnForeignEntries(w io.Writer, module string, foreign, own []string, existingExclude string) { + const maxListed = 20 + fmt.Fprintf(w, "warning: the package holds %d file(s) that belong to other modules, not to %s:\n", len(foreign), module) + for i, name := range foreign { + if i == maxListed { + fmt.Fprintf(w, " ... and %d more\n", len(foreign)-maxListed) + break + } + fmt.Fprintf(w, " %s\n", name) + } + pattern, ambiguous := excludePatternFor(foreign, own) + if pattern != "" { + if existingExclude != "" { + pattern = existingExclude + "|" + pattern + fmt.Fprintln(w, "To leave them out, export again with this --exclude (yours first):") + } else { + fmt.Fprintln(w, "To leave them out, export again with:") + } + fmt.Fprintf(w, " --exclude %s\n", shellQuote(pattern)) + } + if len(ambiguous) > 0 { + fmt.Fprintf(w, "%d of them (e.g. %s) cannot be left out by a pattern that is safe for %s:\n"+ + " it would also match files of the module itself. Remove them from the project's\n"+ + " resources/, javasource/ or themesource/ folder instead.\n", len(ambiguous), ambiguous[0], module) + } +} diff --git a/cmd/mxcli/cmd_marketplace_package_test.go b/cmd/mxcli/cmd_marketplace_package_test.go new file mode 100644 index 000000000..32a3cfa0c --- /dev/null +++ b/cmd/mxcli/cmd_marketplace_package_test.go @@ -0,0 +1,624 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "archive/zip" + "bytes" + "context" + "encoding/json" + "errors" + "io" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + "time" +) + +// packageFake replaces the three seams `marketplace package` calls out through — +// the project reader, mx resolution and the mx process — so the command runs +// without Studio Pro. The mx stub records its arguments and, when exit is 0 and +// entries is non-nil, writes a .mpk holding entries into the directory it was +// given with -d. +type packageFake struct { + version string + modules []string + exit int + output string + entries []string + calls [][]string + mx string + + mxVersion string // version of the mx that resolution finds; defaults to version + mxVersionUnknown bool // the version of that mx cannot be told + gotMxbuildPath string // the --mxbuild-path resolution was given + inspectCopy func(workDir string) +} + +func installPackageFake(t *testing.T, f *packageFake) { + t.Helper() + origInfo, origResolve, origMx, origVer := packageProjectInfo, resolveMxForPackage, mxCreateModulePackageCmd, mxBinaryVersion + t.Cleanup(func() { + packageProjectInfo, resolveMxForPackage, mxCreateModulePackageCmd, mxBinaryVersion = origInfo, origResolve, origMx, origVer + }) + if f.version == "" { + f.version = "11.12.5" + } + if f.mxVersion == "" { + f.mxVersion = f.version + } + packageProjectInfo = func(string) (string, []string, error) { return f.version, f.modules, nil } + resolveMxForPackage = func(mxbuildPath, version string) (string, error) { + f.gotMxbuildPath = mxbuildPath + f.mx = "mx-stub@" + version + return f.mx, nil + } + mxBinaryVersion = func(string) string { + if f.mxVersionUnknown { + return "" + } + return f.mxVersion + } + mxCreateModulePackageCmd = func(_ context.Context, _ string, args []string, out io.Writer) (int, error) { + f.calls = append(f.calls, args) + if f.inspectCopy != nil { + f.inspectCopy(filepath.Dir(args[len(args)-2])) // args end with + } + io.WriteString(out, f.output) + if f.exit != 0 { + return f.exit, nil + } + if f.entries != nil { + dir := pkgArgAfter(args, "-d") + module := args[len(args)-1] + pkgWriteZip(t, filepath.Join(dir, module+".mpk"), f.entries) + } + return 0, nil + } +} + +func pkgArgAfter(args []string, flag string) string { + for i, a := range args { + if a == flag && i+1 < len(args) { + return args[i+1] + } + } + return "" +} + +func pkgWriteZip(t *testing.T, path string, names []string) { + t.Helper() + f, err := os.Create(path) + if err != nil { + t.Fatal(err) + } + zw := zip.NewWriter(f) + for _, n := range names { + w, err := zw.Create(n) + if err != nil { + t.Fatal(err) + } + if !strings.HasSuffix(n, "/") { + w.Write([]byte("x")) + } + } + if err := zw.Close(); err != nil { + t.Fatal(err) + } + f.Close() +} + +// pkgPlaceholderProject writes an .mpr placeholder: the command only copies and +// names it, the reader is faked. +func pkgPlaceholderProject(t *testing.T) string { + t.Helper() + dir := t.TempDir() + p := filepath.Join(dir, "App.mpr") + if err := os.WriteFile(p, []byte("not a real mpr"), 0o644); err != nil { + t.Fatal(err) + } + return p +} + +func runPackageForTest(t *testing.T, opts packageOptions) (stdout, stderr string, res *packageResult, err error) { + t.Helper() + var so, se bytes.Buffer + res, err = runPackage(context.Background(), opts, &so, &se) + return so.String(), se.String(), res, err +} + +var pkgOwnEntries = []string{"project.mpr", "package.xml", "manifest.json", "themesource/pbxexport/web/main.scss"} + +func pkgContainsArg(list []string, s string) bool { + for _, x := range list { + if x == s { + return true + } + } + return false +} + +func TestPackage_DefaultArguments(t *testing.T) { + f := &packageFake{modules: []string{"System", "PbxExport"}, entries: pkgOwnEntries} + installPackageFake(t, f) + mpr := pkgPlaceholderProject(t) + out := filepath.Join(t.TempDir(), "new", "dir") // does not exist yet + + _, _, res, err := runPackageForTest(t, packageOptions{ProjectPath: mpr, Module: "pbxexport", OutputDir: out}) + if err != nil { + t.Fatal(err) + } + if len(f.calls) != 1 { + t.Fatalf("mx called %d times, want 1", len(f.calls)) + } + args := f.calls[0] + if args[0] != "create-module-package" { + t.Errorf("args[0] = %q, want create-module-package", args[0]) + } + if !pkgContainsArg(args, "-l") { + t.Errorf("-l (filter required libs) must be the default: %v", args) + } + if pkgContainsArg(args, "-e") || pkgContainsArg(args, "-m") { + t.Errorf("-e/-m must be absent by default: %v", args) + } + if got := args[len(args)-1]; got != "PbxExport" { + t.Errorf("module argument = %q, want the project's spelling PbxExport", got) + } + target := args[len(args)-2] + if target == mpr || filepath.Base(target) != "App.mpr" { + t.Errorf("mx must run on a temporary copy of the project, got target %q (original %q)", target, mpr) + } + if f.mx != "mx-stub@11.12.5" { + t.Errorf("mx resolved as %q, want the project's version", f.mx) + } + want := filepath.Join(out, "PbxExport.mpk") + if res.Path != want { + t.Errorf("Path = %q, want %q", res.Path, want) + } + if _, err := os.Stat(want); err != nil { + t.Errorf("package not at the output dir: %v", err) + } + if res.Files != len(pkgOwnEntries) { + t.Errorf("Files = %d, want %d", res.Files, len(pkgOwnEntries)) + } + if left, _ := filepath.Glob(filepath.Join(out, ".mxcli-package-*")); len(left) != 0 { + t.Errorf("staging directory left behind: %v", left) + } +} + +func TestPackage_FlagsMapToMxOptions(t *testing.T) { + f := &packageFake{modules: []string{"PbxExport"}, entries: pkgOwnEntries} + installPackageFake(t, f) + _, _, _, err := runPackageForTest(t, packageOptions{ + ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir(), + AllLibs: true, Exclude: "^resources.EventLog.|^themesource.foo.", ExcludeManaged: true, + }) + if err != nil { + t.Fatal(err) + } + args := f.calls[0] + if pkgContainsArg(args, "-l") { + t.Errorf("--all-libs must drop -l: %v", args) + } + if got := pkgArgAfter(args, "-e"); got != "^resources.EventLog.|^themesource.foo." { + t.Errorf("-e = %q", got) + } + if !pkgContainsArg(args, "-m") { + t.Errorf("--exclude-managed-dependencies must pass -m: %v", args) + } +} + +func TestPackage_MapsMxExitCodes(t *testing.T) { + cases := []struct { + exit int + want string + }{ + {1, "project file"}, + {2, "output folder"}, + {3, "module name"}, + {4, "unknown export error"}, + {9, "exit status 9"}, + } + for _, tc := range cases { + f := &packageFake{modules: []string{"PbxExport"}, exit: tc.exit, output: "MX SAYS NO"} + installPackageFake(t, f) + _, _, _, err := runPackageForTest(t, packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir()}) + if err == nil { + t.Fatalf("exit %d: no error", tc.exit) + } + if !strings.Contains(err.Error(), tc.want) { + t.Errorf("exit %d: error %q does not mention %q", tc.exit, err, tc.want) + } + if !strings.Contains(err.Error(), "MX SAYS NO") { + t.Errorf("exit %d: error must carry mx's output: %q", tc.exit, err) + } + } +} + +func TestPackage_UnknownModuleListsModulesAndSkipsMx(t *testing.T) { + f := &packageFake{modules: []string{"System", "Zeta", "Alpha"}} + installPackageFake(t, f) + _, _, _, err := runPackageForTest(t, packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "Typo", OutputDir: t.TempDir()}) + if err == nil { + t.Fatal("want an error for an unknown module") + } + for _, want := range []string{`"Typo"`, "Alpha, System, Zeta"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error %q lacks %q", err, want) + } + } + if len(f.calls) != 0 { + t.Errorf("mx must not run for an unknown module, ran %d time(s)", len(f.calls)) + } +} + +func TestPackage_ExitZeroWithoutPackageIsAnError(t *testing.T) { + // A defensive check: the mx measured (11.12.5) exits 1 for a pattern it cannot + // parse, but a build that exited 0 without a package must not pass for success. + f := &packageFake{modules: []string{"PbxExport"}, output: "Usage: mx create-module-package"} + installPackageFake(t, f) + _, _, _, err := runPackageForTest(t, packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir()}) + if err == nil || !strings.Contains(err.Error(), "wrote no .mpk") || !strings.Contains(err.Error(), "Usage: mx create-module-package") { + t.Fatalf("want a 'wrote no .mpk' error carrying mx's output, got %v", err) + } +} + +func TestPackage_WarnsAboutForeignEntries(t *testing.T) { + f := &packageFake{modules: []string{"PbxExport"}, entries: append([]string{ + "resources/EventLog/x.txt", "resources/EventLog/y.txt", "javasource/other/Foo.java", + }, pkgOwnEntries...)} + installPackageFake(t, f) + stdout, stderr, res, err := runPackageForTest(t, packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir()}) + if err != nil { + t.Fatalf("foreign entries are a warning, not an error: %v", err) + } + if len(res.ForeignEntries) != 3 { + t.Errorf("ForeignEntries = %v", res.ForeignEntries) + } + for _, want := range []string{"resources/EventLog/x.txt", "javasource/other/Foo.java", `--exclude '^javasource.other.|^resources.EventLog.'`} { + if !strings.Contains(stderr, want) { + t.Errorf("stderr lacks %q:\n%s", want, stderr) + } + } + if strings.Contains(stdout, "warning") { + t.Errorf("warnings belong on stderr, stdout was:\n%s", stdout) + } +} + +func TestPackage_JSONShape(t *testing.T) { + f := &packageFake{modules: []string{"PbxExport"}, entries: append([]string{"resources/EventLog/x.txt"}, pkgOwnEntries...)} + installPackageFake(t, f) + var so bytes.Buffer + res, err := runPackage(context.Background(), packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir(), JSON: true}, &so, io.Discard) + if err != nil { + t.Fatal(err) + } + var got map[string]any + if err := json.Unmarshal(so.Bytes(), &got); err != nil { + t.Fatalf("stdout is not a single JSON document: %v\n%s", err, so.String()) + } + if len(got) != 4 { + t.Errorf("keys = %v, want path, sizeBytes, files, foreignEntries", got) + } + if got["path"] != res.Path || got["files"] != float64(5) || got["sizeBytes"].(float64) <= 0 { + t.Errorf("unexpected JSON: %v", got) + } + if fe, _ := got["foreignEntries"].([]any); len(fe) != 1 || fe[0] != "resources/EventLog/x.txt" { + t.Errorf("foreignEntries = %v", got["foreignEntries"]) + } +} + +func TestPackage_JSONForeignEntriesIsEmptyArrayNotNull(t *testing.T) { + f := &packageFake{modules: []string{"PbxExport"}, entries: pkgOwnEntries} + installPackageFake(t, f) + var so bytes.Buffer + if _, err := runPackage(context.Background(), packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir(), JSON: true}, &so, io.Discard); err != nil { + t.Fatal(err) + } + if !strings.Contains(so.String(), `"foreignEntries": []`) { + t.Errorf("want an empty array, got:\n%s", so.String()) + } +} + +func TestPackage_ReplacesAnExistingPackage(t *testing.T) { + f := &packageFake{modules: []string{"PbxExport"}, entries: pkgOwnEntries} + installPackageFake(t, f) + out := t.TempDir() + if err := os.WriteFile(filepath.Join(out, "PbxExport.mpk"), []byte("stale"), 0o644); err != nil { + t.Fatal(err) + } + res, err := runPackage(context.Background(), packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: out}, io.Discard, io.Discard) + if err != nil { + t.Fatal(err) + } + if res.Files != len(pkgOwnEntries) { + t.Errorf("reported a stale package: %+v", res) + } +} + +func TestPackage_StaleFileIsNotReportedWhenMxWritesNothing(t *testing.T) { + f := &packageFake{modules: []string{"PbxExport"}} // exit 0, writes nothing + installPackageFake(t, f) + out := t.TempDir() + pkgWriteZip(t, filepath.Join(out, "PbxExport.mpk"), pkgOwnEntries) + _, _, _, err := runPackageForTest(t, packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: out}) + if err == nil { + t.Fatal("an old package in the output dir must not pass for a fresh export") + } +} + +func TestClassifyPackageEntries_ForeignEntries(t *testing.T) { + p := filepath.Join(t.TempDir(), "m.mpk") + pkgWriteZip(t, p, []string{ + "project.mpr", "package.xml", + "resources/", "resources/loose.txt", // a directory entry; a file directly under resources/ belongs to no module + "resources/PbxExport/own.txt", "resources/pbxexport/own2.txt", // own, in any case + "resources/EventLog/a.txt", "resources\\Win\\b.txt", // backslash separators are normalised + "javasource/pbxexport/Own.java", "javasource/other/Foo.java", + "themesource/pbxexport/web/main.scss", "themesource/atlas_core/web/x.scss", + "userlib/foo.jar", // not attributable to a module: never flagged + }) + names, err := readPackageEntries(p) + if err != nil { + t.Fatal(err) + } + if len(names) != 12 { // 13 entries minus the directory entry + t.Errorf("files = %d, want 12", len(names)) + } + foreign, _ := classifyPackageEntries(names, "PbxExport") + want := []string{"javasource/other/Foo.java", "resources/EventLog/a.txt", "resources/Win/b.txt", "themesource/atlas_core/web/x.scss"} + if !reflect.DeepEqual(foreign, want) { + t.Errorf("foreign = %v\nwant %v", foreign, want) + } +} + +func TestExcludePattern_PlainDotForm(t *testing.T) { + pat, ambiguous := excludePatternFor( + []string{"resources/EventLog/a.txt", "resources/EventLog/b.txt", "themesource/atlas_core/web/x.scss", "resources/My Folder/c.txt"}, + pkgOwnEntries) + if pat != "^resources.EventLog.|^resources.My.Folder.|^themesource.atlas_core." || len(ambiguous) != 0 { + t.Errorf("pattern = %q ambiguous = %v", pat, ambiguous) + } + if strings.ContainsAny(pat, `[]\`) { + t.Errorf("mx rejects character classes and backslashes in -e: %q", pat) + } +} + +func TestExcludePattern_NeverDropsTheModulesOwnFiles(t *testing.T) { + // Exporting EventLog, with a foreign "Event" folder: '^resources.Event.' would + // also match resources/EventLog/..., because '.' stands for any character. + pat, ambiguous := excludePatternFor( + []string{"resources/Event/a.txt"}, + []string{"resources/EventLog/own.txt"}) + if pat != "" || !reflect.DeepEqual(ambiguous, []string{"resources/Event/a.txt"}) { + t.Errorf("pattern = %q ambiguous = %v, want the clash reported and no pattern", pat, ambiguous) + } +} + +func TestPackage_RefusesAnMxOfAnotherVersion(t *testing.T) { + // docker.ResolveMxForVersion prefers an mx on PATH and falls back to any cached + // one, so the mx it returns may belong to another Mendix version than the app. + f := &packageFake{modules: []string{"PbxExport"}, version: "11.12.5", mxVersion: "11.12.1", entries: pkgOwnEntries} + installPackageFake(t, f) + out := t.TempDir() + _, _, _, err := runPackageForTest(t, packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: out}) + if err == nil { + t.Fatal("an mx of Mendix 11.12.1 made a package for a 11.12.5 app without complaint") + } + for _, want := range []string{"11.12.1", "11.12.5", "mxcli setup mxbuild --version 11.12.5", "--mxbuild-path"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should contain %q:\n%v", want, err) + } + } + if len(f.calls) != 0 { + t.Errorf("mx ran %d time(s) although it is the wrong version", len(f.calls)) + } + if left, _ := filepath.Glob(filepath.Join(out, "*")); len(left) != 0 { + t.Errorf("a refused export left %v behind", left) + } +} + +func TestPackage_AcceptsAnMxOfTheProjectsVersion(t *testing.T) { + cases := []struct{ project, mx string }{ + {"11.12.5", "11.12.5"}, + {"10.24.9", "10.24.9.81004"}, // Studio Pro folders carry the build number + {"10.24.9.81004", "10.24.9"}, // and so may the project's stamp + {"11.12.5.1234", "11.12.5.999"}, // same release, other build + } + for _, tc := range cases { + f := &packageFake{modules: []string{"PbxExport"}, version: tc.project, mxVersion: tc.mx, entries: pkgOwnEntries} + installPackageFake(t, f) + _, stderr, _, err := runPackageForTest(t, packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir()}) + if err != nil { + t.Errorf("project %s, mx %s: %v", tc.project, tc.mx, err) + } + if strings.Contains(stderr, "warning") { + t.Errorf("project %s, mx %s: unexpected warning:\n%s", tc.project, tc.mx, stderr) + } + } +} + +func TestPackage_RefusesAnMxWhoseVersionIsUnknown(t *testing.T) { + f := &packageFake{modules: []string{"PbxExport"}, mxVersionUnknown: true, entries: pkgOwnEntries} + installPackageFake(t, f) + _, _, _, err := runPackageForTest(t, packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir()}) + if err == nil || !strings.Contains(err.Error(), "--mxbuild-path") || !strings.Contains(err.Error(), "11.12.5") { + t.Fatalf("want a refusal naming the project's version and the override, got %v", err) + } + if len(f.calls) != 0 { + t.Error("mx ran although its version could not be told") + } +} + +func TestPackage_ExplicitMxbuildPathOverridesButWarns(t *testing.T) { + mxbuildDir := t.TempDir() + for _, tc := range []struct { + name string + unknown bool + mx string + want []string + }{ + {"mismatch", false, "11.12.1", []string{"warning", "11.12.1", "11.12.5"}}, + {"unknown", true, "", []string{"warning", "cannot tell", "11.12.5"}}, + } { + f := &packageFake{modules: []string{"PbxExport"}, mxVersion: tc.mx, mxVersionUnknown: tc.unknown, entries: pkgOwnEntries} + installPackageFake(t, f) + stdout, stderr, res, err := runPackageForTest(t, packageOptions{ + ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir(), MxBuildPath: mxbuildDir, JSON: true}) + if err != nil || res == nil { + t.Errorf("%s: an explicit --mxbuild-path is the override, got %v", tc.name, err) + continue + } + if f.gotMxbuildPath != mxbuildDir { + t.Errorf("%s: resolution got --mxbuild-path %q, want %q", tc.name, f.gotMxbuildPath, mxbuildDir) + } + for _, w := range tc.want { + if !strings.Contains(stderr, w) { + t.Errorf("%s: stderr should contain %q:\n%s", tc.name, w, stderr) + } + } + if strings.Contains(stdout, "warning") || !json.Valid([]byte(stdout)) { + t.Errorf("%s: the warning must stay off stdout, which carries the JSON:\n%s", tc.name, stdout) + } + if len(f.calls) != 1 { + t.Errorf("%s: mx should have run once, ran %d times", tc.name, len(f.calls)) + } + } +} + +func TestPackage_ExplicitMxbuildPathThatDoesNotExistIsAnError(t *testing.T) { + // ResolveMxForVersion silently falls back to PATH when the explicit path is + // bad; the override must not turn a typo into "any mx on PATH, warned about". + f := &packageFake{modules: []string{"PbxExport"}, entries: pkgOwnEntries} + installPackageFake(t, f) + _, _, _, err := runPackageForTest(t, packageOptions{ + ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir(), + MxBuildPath: filepath.Join(t.TempDir(), "no-such-mendix")}) + if err == nil || !strings.Contains(err.Error(), "--mxbuild-path") { + t.Fatalf("want an error naming --mxbuild-path, got %v", err) + } + if len(f.calls) != 0 { + t.Error("mx ran") + } +} + +func TestPackage_MxOutputOnSuccessGoesToStderr(t *testing.T) { + f := &packageFake{modules: []string{"PbxExport"}, output: "MX WARNING: something to read\n", entries: pkgOwnEntries} + installPackageFake(t, f) + stdout, stderr, _, err := runPackageForTest(t, packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir(), JSON: true}) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(stderr, "MX WARNING: something to read") { + t.Errorf("mx's output never reached the user:\nstderr: %q", stderr) + } + if strings.Contains(stdout, "MX WARNING") || !json.Valid([]byte(stdout)) { + t.Errorf("mx's output must not go to stdout, which carries the JSON:\n%s", stdout) + } +} + +func TestPackage_NoMxOutputMeansNothingOnStderr(t *testing.T) { + f := &packageFake{modules: []string{"PbxExport"}, entries: pkgOwnEntries} + installPackageFake(t, f) + _, stderr, _, err := runPackageForTest(t, packageOptions{ProjectPath: pkgPlaceholderProject(t), Module: "PbxExport", OutputDir: t.TempDir()}) + if err != nil || stderr != "" { + t.Errorf("err = %v, stderr = %q", err, stderr) + } +} + +// TestMxCreateModulePackageCmd_StopsWhenContextIsCancelled runs the real process +// runner (not the seam) on this test binary acting as a child that sleeps. +func TestMxCreateModulePackageCmd_StopsWhenContextIsCancelled(t *testing.T) { + if os.Getenv("MXCLI_TEST_SLEEP_CHILD") == "1" { + time.Sleep(time.Minute) + return + } + t.Setenv("MXCLI_TEST_SLEEP_CHILD", "1") + ctx, cancel := context.WithTimeout(context.Background(), 300*time.Millisecond) + defer cancel() + start := time.Now() + var out bytes.Buffer + code, err := mxCreateModulePackageCmd(ctx, os.Args[0], []string{"-test.run=^TestMxCreateModulePackageCmd_StopsWhenContextIsCancelled$"}, &out) + if !errors.Is(err, context.DeadlineExceeded) { + t.Errorf("code = %d, err = %v, want the context's error", code, err) + } + if time.Since(start) > 20*time.Second { + t.Errorf("mx was not stopped by the context (took %s)", time.Since(start)) + } +} + +func TestShellQuote(t *testing.T) { + cases := map[string]string{ + `^resources.EventLog.`: `'^resources.EventLog.'`, + ``: `''`, + `it's`: `'it'\''s'`, + `'a'|'b'`: `''\''a'\''|'\''b'\'''`, + } + for in, want := range cases { + if got := shellQuote(in); got != want { + t.Errorf("shellQuote(%q) = %s, want %s", in, got, want) + } + } +} + +func TestWarnForeignEntries_QuotesAUsersQuoteSafelyForPasting(t *testing.T) { + var w bytes.Buffer + warnForeignEntries(&w, "PbxExport", []string{"resources/EventLog/a.txt"}, pkgOwnEntries, `^resources.it's`) + want := `--exclude '^resources.it'\''s|^resources.EventLog.'` + if !strings.Contains(w.String(), want) { + t.Errorf("want %s in:\n%s", want, w.String()) + } +} + +func TestPackage_OutputFolderInsideTheProjectIsNotCopied(t *testing.T) { + dir := t.TempDir() + mpr := filepath.Join(dir, "App.mpr") + os.WriteFile(mpr, []byte("not a real mpr"), 0o644) + os.MkdirAll(filepath.Join(dir, "dist"), 0o755) + os.WriteFile(filepath.Join(dir, "dist", "Old.mpk"), []byte("earlier package"), 0o644) + os.MkdirAll(filepath.Join(dir, "widgets"), 0o755) + os.WriteFile(filepath.Join(dir, "widgets", "w.mpk"), []byte("a widget, mx needs it"), 0o644) + + var distCopied, widgetCopied bool + f := &packageFake{modules: []string{"PbxExport"}, entries: pkgOwnEntries, inspectCopy: func(work string) { + _, e1 := os.Stat(filepath.Join(work, "dist", "Old.mpk")) + _, e2 := os.Stat(filepath.Join(work, "widgets", "w.mpk")) + distCopied, widgetCopied = e1 == nil, e2 == nil + }} + installPackageFake(t, f) + res, err := runPackage(context.Background(), packageOptions{ProjectPath: mpr, Module: "PbxExport", OutputDir: filepath.Join(dir, "dist")}, io.Discard, io.Discard) + if err != nil { + t.Fatal(err) + } + if distCopied { + t.Error("the output folder's earlier packages were copied along with the project") + } + if !widgetCopied { + t.Error("the rest of the project must still be copied") + } + if _, err := os.Stat(res.Path); err != nil { + t.Errorf("package missing: %v", err) + } +} + +func TestPackage_OutputFolderIsTheProjectFolderStillCopiesTheProject(t *testing.T) { + // "-o ." run from inside the project: skipping the output folder must never + // skip the project itself. + dir := t.TempDir() + mpr := filepath.Join(dir, "App.mpr") + os.WriteFile(mpr, []byte("not a real mpr"), 0o644) + var mprCopied bool + f := &packageFake{modules: []string{"PbxExport"}, entries: pkgOwnEntries, inspectCopy: func(work string) { + _, err := os.Stat(filepath.Join(work, "App.mpr")) + mprCopied = err == nil + }} + installPackageFake(t, f) + if _, err := runPackage(context.Background(), packageOptions{ProjectPath: mpr, Module: "PbxExport", OutputDir: dir}, io.Discard, io.Discard); err != nil { + t.Fatal(err) + } + if !mprCopied { + t.Error("the project was not copied when the output folder is the project folder") + } +} diff --git a/cmd/mxcli/docker/check_copy.go b/cmd/mxcli/docker/check_copy.go index b76d70c80..bc138c9a1 100644 --- a/cmd/mxcli/docker/check_copy.go +++ b/cmd/mxcli/docker/check_copy.go @@ -27,7 +27,7 @@ import ( // Only what mx reads is copied: build output, caches and VCS metadata are skipped // (copyProjectTree), so the cost is the model plus the widget, theme and source // folders. cleanup removes the copy; it is never nil and safe to defer. -func copyProjectToTemp(mprPath string) (workMpr string, cleanup func(), err error) { +func copyProjectToTemp(mprPath string, skip ...string) (workMpr string, cleanup func(), err error) { cleanup = func() {} abs, err := filepath.Abs(mprPath) if err != nil { @@ -46,7 +46,7 @@ func copyProjectToTemp(mprPath string) (workMpr string, cleanup func(), err erro projectDir := filepath.Dir(abs) dst := filepath.Join(tmp, filepath.Base(projectDir)) - if err := copyProjectTree(projectDir, dst); err != nil { + if err := copyProjectTree(projectDir, dst, skip...); err != nil { cleanup() return "", func() {}, err } @@ -73,10 +73,18 @@ var checkCopySkipAnywhere = map[string]bool{ "node_modules": true, } -// copyProjectTree copies src to dst, skipping the folders above. A symlinked -// file is copied as a file, so a tool writing to the copy cannot write through -// it into the original; a symlinked folder is recreated as a link. -func copyProjectTree(src, dst string) error { +// copyProjectTree copies src to dst, skipping the folders above and any folder +// named in skip (compared as the same directory, so spelling and case do not +// matter; src itself is never skipped). A symlinked file is copied as a file, so a +// tool writing to the copy cannot write through it into the original; a symlinked +// folder is recreated as a link. +func copyProjectTree(src, dst string, skip ...string) error { + var skipInfos []fs.FileInfo + for _, s := range skip { + if info, err := os.Stat(s); err == nil && info.IsDir() { + skipInfos = append(skipInfos, info) + } + } return filepath.WalkDir(src, func(p string, d fs.DirEntry, err error) error { if err != nil { return err @@ -94,6 +102,15 @@ func copyProjectTree(src, dst string) error { if rel != "." && (checkCopySkipAnywhere[d.Name()] || (!strings.ContainsRune(rel, filepath.Separator) && checkCopySkipRoot[rel])) { return filepath.SkipDir } + if rel != "." && len(skipInfos) > 0 { + if info, err := d.Info(); err == nil { + for _, s := range skipInfos { + if os.SameFile(info, s) { + return filepath.SkipDir + } + } + } + } return os.MkdirAll(target, 0o755) } if d.Type()&fs.ModeSymlink != 0 { @@ -167,3 +184,29 @@ func (r *pathRewriter) rewrite(s string) string { } return s } + +// WithProjectCopy runs fn with the path of a temporary copy of the project that +// holds mprPath, and removes the copy afterwards. It is for callers that run an +// mx command which writes into the project it is given (theme-cache/, +// deployment/sass/, a module's themesource/ folder) but must leave the user's +// project alone — the same copy Check and MxCheckOnCopy use. +// +// Folders named in skip are left out of the copy — an output folder inside the +// project, whose earlier contents mx has no use for. Naming the project's own +// folder skips nothing. +// +// Output naming the copy should be reported against the project: replace +// filepath.Dir(workMpr) with the project's directory. +func WithProjectCopy(mprPath string, fn func(workMpr string) error, skip ...string) error { + abs, err := filepath.Abs(mprPath) + if err != nil { + return err + } + workMpr, cleanup, err := copyProjectToTemp(abs, skip...) + if err != nil { + return fmt.Errorf("copy the project to a temporary directory: %w\n"+ + " set TMPDIR to a disk with room for the project", err) + } + defer cleanup() + return fn(workMpr) +} diff --git a/cmd/mxcli/docker/check_readonly_test.go b/cmd/mxcli/docker/check_readonly_test.go index 14d92f9fe..bb7d0ce91 100644 --- a/cmd/mxcli/docker/check_readonly_test.go +++ b/cmd/mxcli/docker/check_readonly_test.go @@ -273,3 +273,82 @@ func TestCopyProjectForCheck_MissingProject(t *testing.T) { t.Error("want an error for a project file that does not exist") } } + +// TestWithProjectCopy_WritesStayOnTheCopy: fn gets a copy it may write into, the +// original is untouched, and the copy is gone once fn returns. +func TestWithProjectCopy_WritesStayOnTheCopy(t *testing.T) { + proj := t.TempDir() + mpr := filepath.Join(proj, "App.mpr") + if err := os.WriteFile(mpr, []byte("model"), 0o644); err != nil { + t.Fatal(err) + } + var seen string + err := WithProjectCopy(mpr, func(work string) error { + seen = work + if work == mpr { + t.Error("fn was handed the original project") + } + return os.WriteFile(filepath.Join(filepath.Dir(work), "theme-cache.txt"), []byte("mx wrote this"), 0o644) + }) + if err != nil { + t.Fatalf("WithProjectCopy: %v", err) + } + if _, err := os.Stat(filepath.Join(proj, "theme-cache.txt")); err == nil { + t.Error("a write into the copy reached the original project") + } + if _, err := os.Stat(seen); err == nil { + t.Errorf("the copy %s was not removed", seen) + } + if err := WithProjectCopy(filepath.Join(proj, "nope.mpr"), func(string) error { return nil }); err == nil { + t.Error("want an error for a project file that does not exist") + } +} + +// TestWithProjectCopy_SkipsNamedFoldersButNeverTheProject: a folder inside the +// project that the caller names is left out of the copy; naming the project's own +// folder (an output directory of ".") skips nothing. +func TestWithProjectCopy_SkipsNamedFoldersButNeverTheProject(t *testing.T) { + proj := t.TempDir() + mpr := filepath.Join(proj, "App.mpr") + for p, content := range map[string]string{ + "App.mpr": "model", + "dist/Old.mpk": "earlier package", + "widgets/w.mpk": "a widget", + "userlib/dist/ok.jar": "a deeper folder of the same name", + } { + full := filepath.Join(proj, filepath.FromSlash(p)) + if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(full, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + exists := func(work, rel string) bool { + _, err := os.Stat(filepath.Join(filepath.Dir(work), filepath.FromSlash(rel))) + return err == nil + } + + err := WithProjectCopy(mpr, func(work string) error { + if exists(work, "dist/Old.mpk") { + t.Error("the skipped folder was copied") + } + if !exists(work, "widgets/w.mpk") || !exists(work, "userlib/dist/ok.jar") || !exists(work, "App.mpr") { + t.Error("something outside the skipped folder was not copied") + } + return nil + }, filepath.Join(proj, "dist"), filepath.Join(t.TempDir(), "elsewhere")) + if err != nil { + t.Fatal(err) + } + + err = WithProjectCopy(mpr, func(work string) error { + if !exists(work, "App.mpr") || !exists(work, "dist/Old.mpk") { + t.Error("naming the project folder itself must skip nothing") + } + return nil + }, proj) + if err != nil { + t.Fatal(err) + } +} diff --git a/cmd/mxcli/docker/mx_version.go b/cmd/mxcli/docker/mx_version.go new file mode 100644 index 000000000..16a680be9 --- /dev/null +++ b/cmd/mxcli/docker/mx_version.go @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "os" + "path/filepath" + "regexp" +) + +// mxDepsVersion finds the version of the mx assembly in the mx.deps.json that +// Studio Pro and the mxbuild download ship beside the binary: the target key is +// "mx/" followed by a "\n" escape (measured on 10.24.9 and 11.12.5). +var mxDepsVersion = regexp.MustCompile(`"mx/(\d+\.\d+\.\d+(?:\.\d+)?)[^"]*"`) + +// MxBinaryVersion reports the Mendix version of the mx binary at mxPath, or "" +// when it cannot be told. +// +// ResolveMxForVersion prefers an mx on PATH over the one for the project's +// version and falls back to any cached one, so the binary it returns is not +// necessarily the version asked for; callers that must not mix versions use this +// to find out. The binary's own mx.deps.json is read first, because it holds for +// any install layout (a PATH entry, a renamed folder). The folder name is the +// fallback — ~/.mxcli/mxbuild//modeler/mx and +// Program Files/Mendix//modeler/mx both carry it — and only when it is a +// dotted version of at least three parts. Symbolic links are followed first. +func MxBinaryVersion(mxPath string) string { + if resolved, err := filepath.EvalSymlinks(mxPath); err == nil { + mxPath = resolved + } + if data, err := os.ReadFile(filepath.Join(filepath.Dir(mxPath), "mx.deps.json")); err == nil { + if m := mxDepsVersion.FindSubmatch(data); m != nil { + return string(m[1]) + } + } + if v := versionFromPath(mxPath); isReleaseVersion(v) { + return v + } + return "" +} + +// SameMendixRelease reports whether two Mendix versions are the same release: +// major, minor and patch equal. The build number a Studio Pro folder or a project +// stamp may add ("10.24.9.81004") is not part of the release. An empty or +// malformed version matches nothing. +func SameMendixRelease(a, b string) bool { + pa, okA := parseVersionParts(a) + pb, okB := parseVersionParts(b) + if !okA || !okB || len(pa) < 3 || len(pb) < 3 { + return false + } + return pa[0] == pb[0] && pa[1] == pb[1] && pa[2] == pb[2] +} + +func isReleaseVersion(v string) bool { + parts, ok := parseVersionParts(v) + return ok && len(parts) >= 3 +} diff --git a/cmd/mxcli/docker/mx_version_test.go b/cmd/mxcli/docker/mx_version_test.go new file mode 100644 index 000000000..df6c81a77 --- /dev/null +++ b/cmd/mxcli/docker/mx_version_test.go @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "os" + "path/filepath" + "testing" +) + +// writeMx makes //modeler/mx(.exe) and, when deps is not empty, the +// mx.deps.json beside it. It returns the path of the binary. +func writeMx(t *testing.T, root, verDir, deps string) string { + t.Helper() + dir := filepath.Join(root, verDir, "modeler") + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + bin := filepath.Join(dir, mxBinaryName()) + if err := os.WriteFile(bin, []byte("binary"), 0o755); err != nil { + t.Fatal(err) + } + if deps != "" { + if err := os.WriteFile(filepath.Join(dir, "mx.deps.json"), []byte(deps), 0o644); err != nil { + t.Fatal(err) + } + } + return bin +} + +// depsFor is the shape of the mx.deps.json Studio Pro ships: the target key names +// the assembly and its version, with a literal "\n" escape after the version +// (measured on 10.24.9 and 11.12.5). +func depsFor(version string) string { + return `{"targets": {".NETCoreApp,Version=v10.0": {"mx/` + version + `\n": {"dependencies": {"Mendix.Common": "9.9.9"}}}}}` +} + +func TestMxBinaryVersion_ReadsMxDepsJSONWhateverTheFolderIsCalled(t *testing.T) { + // A folder named after no version (a PATH install, a renamed copy): the + // deps file still says which Mendix version the binary is. + bin := writeMx(t, t.TempDir(), "tools", depsFor("11.12.5")) + if got := MxBinaryVersion(bin); got != "11.12.5" { + t.Errorf("MxBinaryVersion = %q, want 11.12.5", got) + } +} + +func TestMxBinaryVersion_DepsFileWinsOverTheFolderName(t *testing.T) { + // The folder says 11.6.3, the binary says 11.12.5: believe the binary. + bin := writeMx(t, t.TempDir(), "11.6.3", depsFor("11.12.5")) + if got := MxBinaryVersion(bin); got != "11.12.5" { + t.Errorf("MxBinaryVersion = %q, want 11.12.5", got) + } +} + +func TestMxBinaryVersion_FallsBackToTheVersionFolder(t *testing.T) { + // No deps file (an older Studio Pro): ~/.mxcli/mxbuild//modeler/mx and + // Program Files/Mendix//modeler/mx both name the version. + for _, dir := range []string{"11.6.3", "10.24.9.81004"} { + bin := writeMx(t, t.TempDir(), dir, "") + if got := MxBinaryVersion(bin); got != dir { + t.Errorf("MxBinaryVersion(%s) = %q, want %q", dir, got, dir) + } + } +} + +func TestMxBinaryVersion_UnknownWhenNothingSaysSo(t *testing.T) { + for _, tc := range []struct{ name, dir, deps string }{ + {"no deps file, folder is not a version", "tools", ""}, + {"deps file without an mx entry", "tools", `{"targets": {"x": {"other/1.2.3": {}}}}`}, + {"folder looks like a version but is not a release", "v2", ""}, + {"two-part folder", "11.12", ""}, + } { + bin := writeMx(t, t.TempDir(), tc.dir, tc.deps) + if got := MxBinaryVersion(bin); got != "" { + t.Errorf("%s: MxBinaryVersion = %q, want unknown", tc.name, got) + } + } + if got := MxBinaryVersion(filepath.Join(t.TempDir(), "nowhere", "modeler", "mx")); got != "" { + t.Errorf("a binary that does not exist: %q, want unknown", got) + } +} + +func TestSameMendixRelease(t *testing.T) { + for _, tc := range []struct { + a, b string + want bool + }{ + {"11.12.5", "11.12.5", true}, + {"11.12.5", "11.12.1", false}, + {"11.12.5", "11.6.5", false}, + {"11.12.5", "10.12.5", false}, + {"10.24.9", "10.24.9.81004", true}, // build number is not part of the release + {"10.24.9.81004", "10.24.9.99999", true}, + {"11.12.5", "", false}, + {"", "", false}, // nothing known is never a match + {"11.12", "11.12.5", false}, + {"11.12.x", "11.12.5", false}, + } { + if got := SameMendixRelease(tc.a, tc.b); got != tc.want { + t.Errorf("SameMendixRelease(%q, %q) = %v, want %v", tc.a, tc.b, got, tc.want) + } + } +} diff --git a/docs-site/src/guides/marketplace.md b/docs-site/src/guides/marketplace.md index db36673d8..3048c65bf 100644 --- a/docs-site/src/guides/marketplace.md +++ b/docs-site/src/guides/marketplace.md @@ -104,6 +104,71 @@ Agent Commons 4.2.0 requires Mendix 11.12.2, and the project is 11.12.1 `install` installs exactly the content you name. Its dependencies are neither fetched nor named — read the check errors after each install, which identify what is missing by qualified name. The error count is **not monotonic**, because each new module brings its own unmet dependencies: installing the agent-editor stack into a vanilla 11.12.1 app went 0 → 15 → 0 → 18 → 1 → 22 → 1 → 1. Dependencies include widget content as well as modules (`CE0462 "Could not find widget ..."`). +## Exporting a module (`marketplace package`) + +The reverse of `install --file`: write a module of a project out as a `.mpk`, for a company-standard +module, a CI artifact, or to move a module to another app. + +```bash +mxcli marketplace package MyModule -p app.mpr # ./MyModule.mpk +mxcli marketplace package MyModule -p app.mpr -o ./dist --json +``` + +It wraps Mendix's own `mx create-module-package` at the project's Mendix version, so the package is +the one Studio Pro's *Export module package* makes. The module name is checked against the project +first (a typo lists the modules instead of failing with `mx` exit code 3), and `mx`'s exit codes 1–4 +are reported with what they mean and what `mx` printed. Whatever `mx` prints on a successful export +goes to stderr, so `--json` output stays clean. + +**The `mx` must be the project's Mendix version.** `mx` is found from an `mx` on PATH first, then a +Studio Pro install, then the `mxbuild` cache, falling back to any cached one when the project's version +is missing, so the first match can belong to another Mendix version than the app. mxcli reads the +version of the `mx` it found (from the `mx.deps.json` beside it, else its version-named folder), +compares major.minor.patch with the project's, and **refuses** a mismatch — or an `mx` whose version +cannot be told — naming both versions. The fix is `mxcli setup mxbuild --version ` (take an +older `mx` off PATH, which is used before it). `--mxbuild-path` is the deliberate override: with it +a mismatch only warns on stderr and the export goes on. A path that does not exist is an error. + +| Flag | Purpose | +|---|---| +| `-p, --project` | The project holding the module (required). | +| `-o, --output-dir` | Where `.mpk` goes (default: the current directory; created if missing). | +| `--all-libs` | Bundle every `userlib/` JAR. By default only the module's own required libs go in (`mx -l`): 12 KB against 24 MB on an app with Business Events installed. | +| `--exclude ` | Leave out files whose path inside the package matches. `mx` takes **one** expression, so join alternatives with `\|`; write the folder separator as `.`. | +| `--exclude-managed-dependencies` | Leave out managed dependencies (`mx -m`). | +| `--mxbuild-path` | Take `mx` from this Mendix installation (or `mxbuild`) instead of resolving it by the project's version. Overrides the version check: a mismatch warns instead of refusing. | +| `--json` | Print `{path, sizeBytes, files, foreignEntries}`. | + +**Files of other modules.** `-l` filters only `userlib/` JARs. The app's whole `resources/` folder is +still exported (measured on 11.12.5: another module's `resources/EventLog/` came along; its `javasource/` and +`themesource/` did not). After the export mxcli opens the package and warns on stderr about every `resources//…`, `javasource//…` or +`themesource//…` entry where `X` is not the exported module (the last two are a defensive check), +and prints the pattern that drops them: + +```text +warning: the package holds 1 file(s) that belong to other modules, not to MyModule: + resources/EventLog/x.txt +To leave them out, export again with: + --exclude '^resources.EventLog.' +``` + +The pattern is quoted for a POSIX shell (a `'` in your own `--exclude` is written `'''`), so it can be +pasted as it stands; in PowerShell or `cmd` quote it for that shell. + +The pattern uses `.` for the separator because a character class or a backslash (`[/\]`) makes `mx` +print its usage and write nothing. `.` matches any character, so a pattern that would also match a file +of the module itself is not suggested. + +**The project is not changed.** `mx create-module-package` writes into the project it is given: +measured on a blank 11.12.5 app, one run rewrote `theme-cache/web/theme.compiled.css` and its map, +added `deployment/sass/main.scss`, and created the module's `themesource//` folder; the model +files were untouched. mxcli therefore runs it on a temporary copy, as `docker check` does, and a +before/after listing of the project (sizes and modification times) was identical. An output directory +inside the project (`-o ./dist`) is left out of that copy, so earlier packages are not copied along. + +The package installs back with `marketplace install --file`; the round trip into a fresh 11.12.5 app +checks with 0 errors. + ## Why installs do not use `mx module-import` `mx module-import` rewrites an MPR v2 project as v1. Measured on a blank Mendix 11.12.1 app, a single import turned a 69 KB `.mpr` plus 341 `.mxunit` files into one 14 MB SQLite blob with no `mprcontents/` — and the same was observed independently on 11.13.0, so it is not version-specific: diff --git a/docs-site/src/reference/capabilities.md b/docs-site/src/reference/capabilities.md index 4f7ec1cb0..78e39865d 100644 --- a/docs-site/src/reference/capabilities.md +++ b/docs-site/src/reference/capabilities.md @@ -135,6 +135,7 @@ Everything mxcli can do, organized by use case. | Marketplace browse | `mxcli marketplace search/info/versions` | Discover content; `--min-mendix` compatibility filter | | Marketplace download | `mxcli marketplace download [--version X]` | Fetch a content version's `.mpk` to disk | | Marketplace install | `mxcli marketplace install -p app.mpr` | Widget → `widgets/`; new module → `mx module-import`; existing module reported, not updated | +| Marketplace package | `mxcli marketplace package -p app.mpr [-o dir]` | Export a module as a `.mpk` via `mx create-module-package` (module's own libs only; warns about other modules' files); installs back with `install --file` | ## Known Limitations