add docs about @structured

book/custom_commands.md

@@ -1333,6 +1333,10 @@ help commands | where category == deprecated | select name category command_type
 # => ╰───┴───────┴────────────┴──────────────╯
 ```
 
+Attributes also work on [`extern`](externs.md) declarations. One of them, [`@structured`](/commands/docs/attr_structured.md),
+applies only to `extern`. It lets an external command return data instead of text. See
+[Structured Output](externs.md#structured-output) for examples.
+
 To see other attributes available for use within nushell, check the [command reference](/commands/docs/attr.html)
 or run `help attr` from the nushell REPL.
 

book/externs.md

@@ -58,6 +58,169 @@ export extern "git add" [
 ]
 ```
 
+## Structured Output
+
+Many external commands print machine-readable output when you pass a certain flag, such as `git status --porcelain` or `pip list --format=json`. Getting data from them means remembering that flag and the command that parses its output, every time. The [`@structured`](/commands/docs/attr_structured.md) attribute puts both in the `extern` declaration, so you write them once:
+
+```nu
+@structured [--porcelain] { lines | parse --regex '^(?<status>..) (?<file>.+)$' }
+extern "git status" []
+```
+
+`@structured` takes two arguments:
+
+- A list of arguments that make the command print machine-readable output. Each item in the list is one argument, so `[--porcelain]` adds one argument and `[-o json]` adds two, `-o` and `json`. Quote an item that contains spaces or `|`. Otherwise Nushell splits it into several items.
+- A block that turns that output into data. The block receives the command's output as `$in`.
+
+With this declaration, `git status` still prints git's usual text when you run it by itself:
+
+```nu
+git status
+# => On branch main
+# => Changes to be committed:
+# =>   (use "git restore --staged <file>..." to unstage)
+# =>         new file:   INSTALL.md
+# =>
+# => Changes not staged for commit:
+# =>   (use "git add <file>..." to update what will be committed)
+# =>   (use "git restore <file>..." to discard changes in working directory)
+# =>         modified:   README.md
+# =>
+# => Untracked files:
+# =>   (use "git add <file>..." to include in what will be committed)
+# =>         notes.txt
+```
+
+Pipe it into a Nushell command, and Nushell runs `git status --porcelain` instead and passes the output through the block. You get a table:
+
+```nu
+git status | where status =~ M
+# => ╭───┬────────┬───────────╮
+# => │ # │ status │   file    │
+# => ├───┼────────┼───────────┤
+# => │ 0 │  M     │ README.md │
+# => ╰───┴────────┴───────────╯
+```
+
+You don't need a filter to get the table. When you want all of the data, pipe the command into [`collect`](/commands/docs/collect.md). It's an internal command, so the output gets decoded, and it passes its input through unchanged:
+
+```nu
+git status | collect
+# => ╭───┬────────┬────────────╮
+# => │ # │ status │    file    │
+# => ├───┼────────┼────────────┤
+# => │ 0 │ A      │ INSTALL.md │
+# => │ 1 │  M     │ README.md  │
+# => │ 2 │ ??     │ notes.txt  │
+# => ╰───┴────────┴────────────╯
+```
+
+Saving the output in a variable decodes it too:
+
+```nu
+let changes = git status
+$changes | get file
+# => ╭───┬────────────╮
+# => │ 0 │ INSTALL.md │
+# => │ 1 │ README.md  │
+# => │ 2 │ notes.txt  │
+# => ╰───┴────────────╯
+```
+
+When a command can print JSON, the block only needs [`from json`](/commands/docs/from_json.md):
+
+```nu
+@structured [--format=json] { from json }
+extern "pip list" []
+
+pip list | first 3
+# => ╭───┬───────────┬─────────╮
+# => │ # │   name    │ version │
+# => ├───┼───────────┼─────────┤
+# => │ 0 │ contourpy │ 1.4.0   │
+# => │ 1 │ cycler    │ 0.12.1  │
+# => │ 2 │ fonttools │ 4.66.1  │
+# => ╰───┴───────────┴─────────╯
+```
+
+### When Nushell Decodes the Output
+
+Nushell decides by looking at where the output of the extern goes. It adds the `@structured` arguments and runs the block when Nushell itself uses the output:
+
+- Piped into an internal command: `git status | where status =~ M`, or `git status | collect` to get all of the data
+- Saved in a variable: `let changes = git status`
+- Used in a subexpression: `(git status).file`
+
+In every other case, the command runs exactly as you typed it. That includes output that goes:
+
+- To the terminal: `git status`
+- Through `print`: `git status | print`
+- To a file: `git status o> status.txt` or `git status | save status.txt`
+- Into `complete`, `tee`, or `ignore`: `git status | complete`
+- Into another external command or `run-external`: `git status | ^less`
+- Into the same pipe as stderr: `git status o+e>| str trim`
+
+The same rules apply when the extern is the last command of a custom command or a `do` block. After `def gs [] { git status }`, running `gs` prints git's text, and `gs | length` returns `3`, the number of changed files.
+
+::: tip
+Every internal command counts, including commands that work on text. `git status | lines` sends the decoded table to `lines`, not git's text. To get the text, call the command with the caret sigil, which always runs it as written:
+
+```nu
+^git status | lines | first 1
+# => ╭───┬────────────────╮
+# => │ 0 │ On branch main │
+# => ╰───┴────────────────╯
+```
+
+:::
+
+### Adding Your Own Arguments
+
+Nushell puts the `@structured` arguments right after the command name, before any arguments you type. This extern gives `git log` an output format that is easy to split into columns:
+
+```nu
+@structured ["--format=%h|%an|%s"] { lines | split column "|" hash author subject }
+extern "git log" []
+
+git log | where author == "Ana Silva"
+# => ╭───┬─────────┬───────────┬──────────────────╮
+# => │ # │  hash   │  author   │     subject      │
+# => ├───┼─────────┼───────────┼──────────────────┤
+# => │ 0 │ 73e9a8f │ Ana Silva │ Print a greeting │
+# => │ 1 │ 83c0a31 │ Ana Silva │ Initial commit   │
+# => ╰───┴─────────┴───────────┴──────────────────╯
+```
+
+`git log -n 2 | where author == "Ana Silva"` runs `git log --format=%h|%an|%s -n 2`.
+
+Because your arguments come last, a flag that changes the output format replaces the one from `@structured`. The block then gets output it wasn't written for. Here `--oneline` wins, so each whole line lands in the `hash` column:
+
+```nu
+git log --oneline -n 1 | first
+# => ╭──────┬──────────────────────────╮
+# => │ hash │ 73e9a8f Print a greeting │
+# => ╰──────┴──────────────────────────╯
+```
+
+To use a different format, call the command with the caret sigil, such as `^git log --oneline`.
+
+### More About `@structured`
+
+- The block can read `$env`, constants, and `let` variables defined before the `extern`, just like the body of a `def`. It can't use `mut` variables.
+- `@structured` only works on `extern`. Putting it on a `def` is a parse error.
+- If an `extern` has more than one `@structured`, Nushell uses the last one.
+- When the command fails, its error message is printed. What happens next depends on the block. A block that reads all of the output, such as `from json`, stops the pipeline with a non-zero exit code error that points at the call. A block that streams the output, such as `lines`, passes on whatever the command printed and doesn't raise an error.
+
+`@structured` also works with `export extern`, so a module can share these declarations:
+
+```nu
+module git-data {
+  @structured [--porcelain] { lines | parse --regex '^(?<status>..) (?<file>.+)$' }
+  export extern "git status" []
+}
+use git-data "git status"
+```
+
 ## Limitations
 
 There are a few limitations to the current `extern` syntax. In Nushell, flags and positional arguments are very flexible—flags can precede positional arguments, flags can be mixed into positional arguments, and flags can follow positional arguments. Many external commands are not this flexible. There is not yet a way to require a particular ordering of flags and positional arguments to the style required by the external.

book/running_externals.md

@@ -71,7 +71,7 @@ let program = "git"
 ^$program status
 ```
 
-See [Strings as external commands](working_with_strings.md#strings-as-external-commands) for the supported string forms. An [`extern` declaration](externs.md) can add type checking and completions for a known external command; calling it follows the same argument rules described here.
+See [Strings as external commands](working_with_strings.md#strings-as-external-commands) for the supported string forms. An [`extern` declaration](externs.md) can add type checking and completions for a known external command, and its [`@structured`](externs.md#structured-output) attribute can turn the command's output into data. Calling it follows the same argument rules described here.
 
 ## Additional Windows Notes
 

commands/docs/attr.md

@@ -2,7 +2,7 @@
 title: attr
 categories: |
   core
-version: 0.116.0
+version: 0.116.2
 core: |
   Various attributes for custom commands.
 usage: |
@@ -43,4 +43,5 @@ You must use one of the following subcommands. Using this command as-is will onl
 | [`attr deprecated`](/commands/docs/attr_deprecated.md)               | Attribute for marking a command or flag as deprecated.                        | built-in |
 | [`attr example`](/commands/docs/attr_example.md)                     | Attribute for adding examples to custom commands.                             | built-in |
 | [`attr interactive`](/commands/docs/attr_interactive.md)             | Attribute marking a completer as interactive.                                 | built-in |
-| [`attr search-terms`](/commands/docs/attr_search-terms.md)           | Attribute for adding search terms to custom commands.                         | built-in |
+| [`attr search-terms`](/commands/docs/attr_search-terms.md)           | Attribute for adding search terms to custom commands.                         | built-in |
+| [`attr structured`](/commands/docs/attr_structured.md)               | Attribute for getting structured output from an external command.             | built-in |

commands/docs/attr_structured.md

@@ -0,0 +1,77 @@
+---
+title: attr structured
+categories: |
+  core
+version: 0.116.2
+core: |
+  Attribute for getting structured output from an external command.
+usage: |
+  Attribute for getting structured output from an external command.
+editLink: false
+contributors: false
+---
+<!-- This file is automatically generated. Please edit the command in https://github.com/nushell/nushell instead. -->
+
+# `attr structured` for [core](/commands/categories/core.md)
+
+<div class='command-title'>Attribute for getting structured output from an external command.</div>
+
+## Command Type
+
+`built-in`
+
+## Signature
+
+```> attr structured {flags} (args) (decode)```
+
+## Parameters
+
+ -  `args`: Arguments that make the external command print structured output.
+ -  `decode`: Turns that output, given as `$in`, into data.
+
+
+## Input/output types:
+
+| input   | output                                      |
+| ------- | ------------------------------------------- |
+| nothing | record&lt;args: list&lt;string&gt;, decode: closure&gt; |
+## Examples
+
+Get the changed files from `git status --porcelain` as a table.
+```nu
+> @structured [--porcelain] { lines | parse --regex '^(?<status>..) (?<file>.+)$' }
+    extern "git status" []
+    git status | where status =~ M | get file
+
+```
+
+Get the installed Python packages from `pip list --format=json` as a table.
+```nu
+> @structured [--format=json] { from json }
+    extern "pip list" []
+    pip list | where name starts-with num
+
+```
+
+Get the outdated Homebrew formulae from `brew outdated --json=v2` as a table.
+```nu
+> @structured [--json=v2] { from json | get formulae }
+    extern "brew outdated" []
+    brew outdated | where pinned == false | get name
+
+```
+
+Get pods from kubectl as a table when the output is piped into `where`.
+```nu
+> @structured [-o json] { from json | get items }
+    extern "kubectl get" [resource: string, --namespace(-n): string]: nothing -> table
+    kubectl get pods | where status.phase != Running
+
+```
+
+## Notes
+When the output of the extern is used by nushell (piped into an internal command or collected into a variable), the arguments are added right after the command's name and the output is decoded with the block. When the output goes to the terminal, `print`, a file, `ignore`, `save`, `tee`, `complete` or another external command (`run-external` included), the command runs as written. The same holds when the extern is the last command of a `do` block or a custom command. Calling the command with a caret, like `^git status`, always runs it as written.
+
+The arguments come before the user's own, so a format flag the user passes can override them and break the decode, as in `kubectl get pods -o wide`. Use a caret call for that. `tee` passes the command's own output on to the commands after it. When an extern has more than one `@structured`, the last one is used.
+
+Like a `def` body, the block can read `$in`, `$env`, constants and the `let` variables declared before the extern, but not `mut` variables.