HTML API: Add a helper to split space-separated tokens. by Sabbir345 · Pull Request #13917 · WordPress/wordpress-develop

Sabbir345 · 2026-10-02T06:39:08Z

Trac ticket: https://core.trac.wordpress.org/ticket/65466

Introduces wp_split_space_separated_tokens(), a generator that yields each token of a plain-text string split on ASCII whitespace (" \t\n\f\r"), as a browser does for a set of space-separated tokens such as a list of class names. It then uses the helper to fix class name parsing in block supports.

Notes

Following the discussion on the ticket, the helper is a lexical operation on plain text, not HTML. It does not decode character references, deduplicate tokens, or normalize case, and it is named for space-separated tokens rather than class lists to avoid confusion with parsing an HTML class attribute. For HTML, WP_HTML_Tag_Processor::class_list() remains the right tool.
The helper is a generator, since every caller iterates the tokens in a foreach loop.
The helper lives in functions.php next to wp_parse_list(), which is not suitable here because it also splits on commas.
wp_render_elements_class_name() and wp_render_custom_css_class_name() contained the same strtok() loop. Both now use the helper, with no change in behavior.
The following functions are fixed:
- wp_render_block_style_variation_class_name() used /\bis-style-(\S+?--\d+)\b/. Because \b matches after a hyphen, custom-is-style-fancy--1 and is-style-fancy--1-extra both caused is-style-fancy--1 to be added to the block markup.
- wp_get_block_style_variation_name_from_class() used \b the same way, so it matched is-style-* inside another class name. This is the core counterpart of gutenberg_get_block_style_variation_name_from_class() listed in the ticket.
- wp_get_block_style_variation_name_from_registered_style() used explode( ' ', … ), so a variation separated by a tab or newline was missed.
Existing behavior is otherwise kept. For example, wp_get_block_style_variation_name_from_class() still skips any variation whose name starts with default, as the (?!default) lookahead did.
WP_Duotone::restore_image_outer_container() is not changed. It was already updated to use the HTML API class helpers in r62627, as noted in comment:3.
This takes a different approach from HTML API: Add class_list_from_string() helper to tokenize decoded class names. #12499, which adds a class_list_from_string() method to WP_HTML_Tag_Processor without updating the call sites.
The Gutenberg counterparts (gutenberg_*) will need the same change.

Testing Instructions

Run the new and related tests:
npm run test:php -- --filter 'wpSplitSpaceSeparatedTokens|WpGetBlockStyleVariationNameFromClass|Tests_Block_Supports_Layout|Tests_Block_Supports_BlockStyleVariations|WpRenderElementsSupport|WpRenderCustomCssClassName'
Run the wider groups: npm run test:php -- --group block-supports,functions,blocks.

Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Model(s): Claude Opus 5.5
Used for: Verifying the ticket against trunk, implementing the helper and call-site changes, addressing review feedback, writing the tests, and drafting this description.

This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.

Introduces `wp_split_space_separated_tokens()`, which splits a plain-text string on ASCII whitespace, as a browser does for a set of space-separated tokens such as a list of class names. It performs no character reference decoding, deduplication, or case normalization. The helper replaces duplicated `strtok()` loops in `wp_render_elements_class_name()` and `wp_render_custom_css_class_name()`, and fixes class name parsing in the following functions: - `wp_render_block_style_variation_class_name()` and `wp_get_block_style_variation_name_from_class()` used a `\b` word boundary, which matched `is-style-*` inside another class name such as `custom-is-style-fancy--1`. - `wp_get_block_style_variation_name_from_registered_style()` split only on spaces, missing variations separated by tabs or newlines. Fixes #65466.

github-actions · 2026-10-02T06:39:19Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @RumanCodes.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

Core Committers: Use this line as a base for the props when committing in SVN:

Props sabbir345, jonsurrell.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

github-actions · 2026-10-02T06:42:39Z

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

All changes will be lost when closing a tab with a Playground instance.
All changes will be lost when refreshing the page.
A fresh instance is created each time the link below is clicked.
Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

RumanCodes · 2026-10-06T12:11:42Z

Test report

Environment — WordPress 7.2-alpha (trunk 2bf0714531), PHP 8.3, single site, local
wordpress-develop env. PR head 7658328747.

The helper matches the spec

wp_split_space_separated_tokens() splits on exactly the five ASCII whitespace characters and
nothing else:

Input	Splits?
space, TAB, LF, FF, CR	yes	correct
vertical tab `U+000B`	no	correct — not ASCII whitespace
NUL, NBSP `U+00A0`	no	correct
`''` and `" \t\n "`	0 tokens	correct

Behaviour changes at the call sites

I ran 16 edge cases through wp_get_block_style_variation_name_from_class() on trunk and on
this branch. Four differ:

Input	trunk	this PR
`foo-is-style-blue`	`{blue}`	`{}`	correct — `\b` let the old regex match mid-token; covered by your `prefix inside another class name` test
`foo-is-style-blue--12-bar`	`{blue--12-bar}`	`{}`	same cause
`is-style-blue,`	`{blue}`	`{blue,}`	correct — the token really is `blue,`; covered by your `variation ending in a non-word character` test
`is-style-a` + `U+000B` + `is-style-b`	`{a, b}`	`{ais-style-b}`	correct per spec, and the only one of the four with no test. PCRE `\S` treats vertical tab as a separator; the HTML spec's ASCII whitespace does not. Might be worth a data-set row to pin it

Automated

--group functions,block-supports,formatting: 3823 tests, 0 failures. The 32 warnings
are the pre-existing PHPUnit-10 E_STRICT deprecation notices, present on trunk too.
PHPCS on all five changed source files: 0 errors. The one warning
(functions.php:1858, DESCRIBE $table) is pre-existing on trunk — I verified it there.

The failing CI job looks unrelated

PHP 8.3 / MariaDB 11.8 with memcached has exactly one failure:

1) Tests_Formatting_ConvertSmilies::test_convert_standard_smilies with data set #1
-'... We got fun n games! 🙂 ...'
+'... We got fun n games! :) ...'

There is no call path from that test to wp_split_space_separated_tokens() or to any file this
PR touches, and the test passes locally on this branch inside the formatting group.

A likely mechanism, which I could not reproduce locally — I had no way to enable memcached
in this environment, so treat it as a hypothesis rather than a finding:
convert_smilies() bails when get_option( 'use_smilies' ) is falsy, and
test_convert_standard_smilies() sets use_smilies to 0 inside the test body, relying on
the transaction rollback to put it back. A rollback restores the DB row but not a persistent
object cache, so under memcached data set #1 can read a stale 0 — which is exactly where it
failed, right after data set #0 set it to 0. That would make this an instance of #65893
rather than anything to do with this patch; adding a test file is enough to shift ordering and
expose it.

Suggested next step: re-run just that job. Green means ordering; red again points at
convertSmilies.php, not at this PR.

Not tested

The memcached configuration itself (reason above), multisite, the full suite locally, and the
edge cases of wp_render_block_style_variation_class_name() beyond the two rows in the table —
its lazy-to-greedy change on inputs like is-style-a--1--2 is worth a look.

One note

This is 26 commits behind trunk, so it will want a refresh before commit. And since it adopts
dmsnell's "set of space-separated tokens" framing from comment:4 on the ticket rather than the
earlier class-list naming, a committer nod on the name would probably unblock it — there has
been no review on it yet.

AI assistance: Yes (Claude Code), used for the edge-case comparison, running the suites, triaging
the CI failure and drafting this report; reviewed and tested by me.

sirreal · 2026-10-19T09:15:27Z

+			// @ticket 65466
+			'whitespace separators'                    => array(
+				'class_string' => "is-style-light\tcustom-class\nis-style-outline",
+				'expected'     => array( 'light', 'outline' ),
+			),
+			// @ticket 65466
+			'prefix inside another class name'         => array(
+				'class_string' => 'custom-is-style-outline',
+				'expected'     => array(),
+			),
+			// @ticket 65466
+			'variation ending in a non-word character' => array(
+				'class_string' => 'is-style-outline-',
+				'expected'     => array( 'outline-' ),
+			),
+			// @ticket 65466
+			'default variation'                        => array(
+				'class_string' => 'is-style-default is-style-outline',
+				'expected'     => array( 'outline' ),
+			),


I don't think we need to add tests to all the functions that use this new method for token splitting for more cases.

Thanks! I removed the test cases that were only checking token splitting (like a tab or newline between class names), since the helper's own tests already cover that. I kept the cases that test the actual bug this PR fixes, like 'prefix inside another class name' and 'instance number followed by a suffix', since those are not about token splitting, they check the real logic of these functions. Happy to remove more if you think any of these are still not needed.

sirreal · 2026-10-19T09:24:22Z

+function wp_split_space_separated_tokens( string $input ): array {
+	$tokens     = array();
+	$delimiters = " \t\n\f\r";
+	$length     = strlen( $input );
+	$offset     = strspn( $input, $delimiters );
+
+	while ( $offset < $length ) {
+		$token_length = strcspn( $input, $delimiters, $offset );
+		$tokens[]     = substr( $input, $offset, $token_length );
+		$offset      += $token_length;
+		$offset      += strspn( $input, $delimiters, $offset );
+	}
+
+	return $tokens;
+}


Note that all the consumers are foreach loops. This would be better as a generator yielding the tokens.

Done in b0cc998: it now yields each token. In e0a697a I also aligned the loop and docblock with WP_HTML_Tag_Processor::class_list().

All consumers iterate the tokens in a `foreach` loop, so yield each token instead of building an array. Also drop the call-site test cases that only exercise token splitting, which the helper's own tests already cover. See #65466.

…rated-tokens

…ns()`. The helper's own tests cover token splitting, so the functions that use it do not need additional cases. See #65466.

Use the same loop structure, boundary characters, and documentation style as `WP_HTML_Tag_Processor::class_list()`, and follow the same test conventions: a static data provider and an assertion message. See #65466.

The previous commit dropped these along with the token-splitting duplicates, but they are not about tokenization: they are the regression tests for the \b word-boundary bug this ticket fixes, and are referenced by name in the PR description's testing instructions. See #65466.

Merge branch 'trunk' into fix/65466-space-separated-tokens

7658328

Sabbir345 added 2 commits October 12, 2026 11:59

Merge branch 'trunk' into fix/65466-space-separated-tokens

e319bf8

Merge branch 'trunk' into fix/65466-space-separated-tokens

9904a76

sirreal reviewed Oct 19, 2026

View reviewed changes

Sabbir345 added 5 commits October 20, 2026 06:41

Merge remote-tracking branch 'origin/trunk' into fix/65466-space-sepa…

b1abae2

…rated-tokens

Tests: Remove call-site test cases for `wp_split_space_separated_toke…

99ceb98

…ns()`. The helper's own tests cover token splitting, so the functions that use it do not need additional cases. See #65466.

HTML API: Align wp_split_space_separated_tokens() with class_list().

e0a697a

Use the same loop structure, boundary characters, and documentation style as `WP_HTML_Tag_Processor::class_list()`, and follow the same test conventions: a static data provider and an assertion message. See #65466.

Navigation Menu

Repository navigation

HTML API: Add a helper to split space-separated tokens. - #13917

HTML API: Add a helper to split space-separated tokens.#13917
Sabbir345 wants to merge 9 commits into
WordPress:trunkfrom
Sabbir345:fix/65466-space-separated-tokens

Sabbir345 commented Oct 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

RumanCodes commented Oct 6, 2026

Uh oh!

sirreal Oct 19, 2026

Uh oh!

Sabbir345 Oct 20, 2026

Uh oh!

sirreal Oct 19, 2026

Uh oh!

Sabbir345 Oct 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Sabbir345 commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes

Testing Instructions

Use of AI Tools

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unlinked Accounts

Uh oh!

github-actions Bot commented Oct 2, 2026

Test using WordPress Playground

Some things to be aware of

Uh oh!

RumanCodes commented Oct 6, 2026

Test report

The helper matches the spec

Behaviour changes at the call sites

Automated

The failing CI job looks unrelated

Not tested

One note

Uh oh!

sirreal Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Sabbir345 Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

sirreal Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Sabbir345 Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Sabbir345 commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading