perf: move EG() and CG() in ZTS builds into __thread storage by henderkes · Pull Request #23227 · php/php-src

henderkes · 2026-08-12T12:12:02Z

replay of #22231

Moves EG and CG into __thread storage after all. We first moved them into constant offsets (#22287) from *_tsrm_ls_cache, but I couldn't find a way to stop gcc or clang from reloading _tsrm_ls_cache base pointer between function calls, leading to an extra pointer load once per function.

This eliminates the pointer load, making the access sequence to EG/CG just a single mov (x64) / mrs + add + ldr (aarch64) under local-exec. initial-exec likewise loses the pointer load so 3 -> 2 instructions (x64).
cc @arnaud-lb

What I'm adding here to counter the global-dynamic fallback slowdown is the option to explicitly opt-in to initial-exec, under the knowledge that host programs loading it will need to increase the static tls surplus. This is not an issue for package providers.

PS: Actually figured out that the explicit model choice doesn't happen for musl, not sure about IE (static tls surplus on musl?), but LE should work just fine. That's for another PR though.

arnaud-lb

I like the --with-tsrm-tls-model idea.

Did a first pass, but I will take the time to review carefully.

henderkes

It's a bit of a pain to find these on mobile, so I'll pin just these two, but there's more.

It could eventually be reworked when all the symbols move directly into thread storage, but I've not even begun thinking about it. There's probably no point except for zend_ini_scanner_globals and what we still have in the front.

henderkes · 2026-08-20T16:33:13Z

Created the aarch64 global-dynamic first first, but I may as well look into teaching the JIT to take the address of _tsrm_ls_cache + offset instead of the mandatory load for a bit of JIT speedup.

henderkes · 2026-08-20T16:35:50Z

@arnaud-lb out of

zend_ini_scanner_globals))
virtual_cwd_globals))
zend_signal_globals_t))
zend_gc_globals_size())
php_core_globals))
sapi_globals_struct))
zend_accel_globals))
zend_jit_globals))

Which do you think would make sense moving too? gc globals probably and maybe virtual_cwd_globals?

arnaud-lb · 2026-08-20T16:45:42Z

alloc_globals is likely the most accessed global as every emalloc/efree and related fetch AG(mm_heap). I know that zend_alloc_globals is not exposed, but we could reserve enough space in _tsrm_ls_cache without exposing it:

struct _zend_tsrm_ls_cache {
	void *cache;
	void *self;
	zend_executor_globals eg;
	zend_compiler_globals cg;
	uint8_t ag[123];
	...
};

// zend_alloc.c:
ZEND_STATIC_ASSERT(sizeof(zend_tsrm_ls_cache.ag) >= sizeof(zend_alloc_globals));

#define AG(v) (((zend_alloc_globals*)&_tsrm_ls_cache.ag)->v)

henderkes · 2026-08-20T16:47:51Z

I think I already have a branch open for the AG move... perhaps it makes more sense moving it here, though.

Edit: my memory these days, it's already in. But I just realised we're keeping a useless write around.

henderkes · 2026-08-25T10:06:59Z

Created the aarch64 global-dynamic first first, but I may as well look into teaching the JIT to take the address of _tsrm_ls_cache + offset instead of the mandatory load for a bit of JIT speedup.

Proving too far outside my expertise. I think this is reviewable like this now, the JIT optimization could be done later in IR side to let us drop the self reference.

Unless @dstogov would like to have a go at it.

arnaud-lb

The changes look good in general, but it's a bit hard to follow. Possibly grouping abstractions in TSRM would help? Right now you need to have TSRM.*, zend.c, zend_globals_macros.h to understand what's going on.

The Windows changes make sense, but maybe @shivammathur can take a look as well?

henderkes · 2026-09-04T13:39:15Z

to me central position seems better, although I can see the appeal of trying to keep it inside the module

From @iliaal. I'll try approaching this with global per-thread cleanup hooks, but leave persistent_list in EG and wire ts_free_thread to clean it up.

henderkes · 2026-09-05T11:51:28Z

-	}
+	do {
+		zend_try {
+			zend_destroy_rsrc_list(&EG(persistent_list));


this is technically a bugfix, plist previously leaked on threads that unregistered with ts_free_thread().

henderkes · 2026-09-12T06:47:42Z

@iliaal could you take a look to see if the change looks acceptable to you?

iliaal · 2026-09-13T19:50:36Z

+ZEND_API void zend_win_tsrm_cache_shutdown(void);
+ZEND_API zend_tsrm_ls_cache *zend_win_tsrm_cache_fallback(void);
+#  ifdef __clang__
+static __inline__ __attribute__((const, always_inline)) zend_tsrm_ls_cache *zend_win_tsrm_cache_ptr(void)


const is formally wrong here: the function reads zend_win_tsrm_cache_offset and gs: memory. It is harmless in practice since the published value is thread-invariant, and the CSE it enables is safer than re-reading gs after a TlsFree. __readgsqword under const gives the same instruction on clang-cl, so the clang branch and its asm could go and leave one implementation.

Formally wrong, but safe in our circumstances and it gets clang to cse where it otherwise wouldn't.

Technically we can always &_tsrm_ls_cache, it's just not optimal codegen.

__readgsqword under const gives the same instruction on clang-cl, so the clang branch and its asm could go and leave one implementation.

Not on PC to test this atm but I'm 98% sure it resulted in unnecessary reloads under Clang 21 and 22.

iliaal · 2026-09-13T19:50:37Z

+AS_VAR_IF([PHP_THREAD_SAFETY], [yes], [
+  AS_CASE([$PHP_TSRM_TLS_MODEL],
+    [auto], [
+      AS_VAR_IF([php_tsrm_dlopened_sapi], [yes],


apache2handler is the only SAPI setting this. sapi/embed also builds a libphp.so under --enable-embed=shared, so a host that dlopens it rather than linking against it gets initial-exec and the static TLS surplus failure this flag exists to avoid. Setting it there too, or stating that embed is link-time only, would close the gap.

I'm not aware of any embed users that don't link against it. If there are, I'm happy to declare embed sapi as dlopened too, I can always overwrite the model.

ts_free_resources(), ts_free_thread(), ts_free_id() and ts_apply_for_id() matched entries with p->thread_id == tsrm_thread_id(). That is ambiguous in exactly the case the surrounding code exists to handle: a stale entry of a dead thread can carry the live thread's recycled id. Compare against the entry that tsrm_tls_get() hands out instead. The recycle path in ts_resource_ex() no longer runs the stale entry's destructors. It used to point the TLS cache at an entry whose native TLS block had died with its thread, so any destructor reaching for EG or CG read freed memory. Leaking the dead thread's module globals is the better trade; a child process that recycles thread ids gets respawned by the SAPI anyway. Hold the shutdown marker across the window so that signal handlers stay away from both the stale entry and its replacement. Also reset tls_key, the TSRM tables and TSRMLS_CACHE on shutdown.

…t budget

…free_thread` at thread exit SAPI's should already been doing this on their own before, but the apache2handler doesn't get control over threads, so it didn't call ts_free_thread and therefore started leaking resources with the move of per-thread globals from heap storage to native __thread storage

arnaud-lb

@henderkes I want to merge this but the amount of changes make it extremely hard to review, and the PR grows every time I come back to it. Could you split it?

arnaud-lb · 2026-10-12T09:19:46Z

-      CFLAGS="$MACHINE $MUSL $opt -Werror -I$root/ext/opcache -I$root/Zend -I$root"
+      # def.c stubs tsrm_get_ls_cache_tcb_offset() to 0, which only happens under
+      # global-dynamic, so reference the cache with that model too.
+      CFLAGS="$MACHINE $MUSL $opt -Werror -DTSRM_TLS_MODEL_USE_GLOBAL_DYNAMIC -I$root/ext/opcache -I$root/Zend -I$root"


Unfortunately this nerfs the test suite and hides the reason of the failure.

To give some context, zend_jit_resolve_tsrm_ls_cache_offsets() looks at code that was potentially relaxed by the linker, to find the TLS offsets, but there is at least one test configuration (shared_def_static_user) in which this fails.

Before this branch, the TLS var was not relaxed in that configuration (it used the GD model). Since this branch it uses the IE model.

zend_jit_resolve_tsrm_ls_cache_offsets() only recognizes the GD and LE sequences. We need to add IE support to fix the test (for x64 and aarch64).

I've not overlooked the comment, I'm just taking my time understanding the logic. Will do tomorrow or on Thursday.

Addressed, but now a weird test case is failing: libdef.so linked with initial-exec under NO_SURPLUS defined. That previously just asserted that global-dynamic was selected, but it isn't now for non-dlopened shared libraries, and the libdef.so helper doesn't get -DTSRM_TLS_MODEL_USE_GLOBAL_DYNAMIC anymore, so it gets initial-exec.

What is that test actually good for? Assert that global-dynamic is used? Or that initial-exec is selected, but then it obviously can't load in the NO_SURPLUS case.

arnaud-lb · 2026-10-12T10:33:55Z

+{
+	uintptr_t offset = zend_win_tsrm_cache_offset;
+	zend_tsrm_ls_cache *ptr;
+	if (__builtin_expect(offset == 0, 0)) {


Nit: UNEXPECTED()

can't, zend_portability includes tsrm, so tsrm can't use zend_portability macros.

(meaning, we can, but only by moving the definitions out of zend_portability, which is imo the right place for them. we don't need the conditional here because it's already gated to clang-cl 22)

Why does zend_portability include TSRM? I don't see any obvious reason why this is required.

I have no idea, the commit dates back to 2014 where it was split off from zend.h. I haven't looked further into the history.

As far as I can see it only uses TSRM_TLS to define ZEND_TLS

henderkes · 2026-10-12T16:06:35Z

@henderkes I want to merge this but the amount of changes make it extremely hard to review, and the PR grows every time I come back to it. Could you split it?

Funny you mention this, it seems that 8.6 is already running into static TLS surplus issues, so I was planning on opening a split of the configure arg against 8.6 anyway. See #24262.

I'll see what more I can split off to merge separately.

henderkes requested review from bukka and dstogov as code owners August 12, 2026 12:12

github-actions Bot added Category: Build System Category: Engine Extension: opcache SAPI: apache2handler ABI break labels Aug 12, 2026

arnaud-lb reviewed Aug 12, 2026

View reviewed changes

Comment thread ext/opcache/jit/zend_jit_ir.c Outdated

Comment thread TSRM/TSRM.h Outdated

Comment thread Zend/zend.c Outdated

henderkes force-pushed the perf/1-tls-eg-cg branch from b78839c to 5d9e202 Compare August 13, 2026 19:43

henderkes commented Aug 14, 2026

View reviewed changes

Comment thread TSRM/TSRM.h Outdated

Comment thread TSRM/TSRM.h

henderkes mentioned this pull request Aug 20, 2026

allow global-dynamic on glibc dstogov/ir#181

Merged

henderkes force-pushed the perf/1-tls-eg-cg branch from bb0529c to 3c81898 Compare August 20, 2026 17:03

henderkes commented Aug 20, 2026

View reviewed changes

Comment thread Zend/zend_gc.c

arnaud-lb reviewed Aug 26, 2026

View reviewed changes

Comment thread Zend/zend_globals_macros.h Outdated

Comment thread Zend/zend_globals_macros.h Outdated

Comment thread Zend/zend_globals_macros.h Outdated

Comment thread Zend/zend_globals_macros.h Outdated

Comment thread Zend/zend.c Outdated

shivammathur reviewed Aug 26, 2026

View reviewed changes

Comment thread win32/dllmain.c

Comment thread Zend/zend.c

henderkes force-pushed the perf/1-tls-eg-cg branch from f5c7bcc to 4c08c6c Compare August 28, 2026 16:33

shivammathur reviewed Aug 28, 2026

View reviewed changes

Comment thread TSRM/TSRM.c Outdated

henderkes commented Sep 5, 2026

View reviewed changes

henderkes requested review from arnaud-lb and shivammathur September 5, 2026 11:51

iliaal reviewed Sep 13, 2026

View reviewed changes

Comment thread TSRM/TSRM.c

iliaal reviewed Sep 13, 2026

View reviewed changes

henderkes added 18 commits September 29, 2026 13:15

remove useless line

18c0f65

move GC globals into __thread storage too

bcf1a46

feedback: teb slot needs _m_x64

016e1ab

feedback: precompute teb slot offset

555646d

feedback: move macros into tsrm.h

096ee6b

feedback: add zend_tsrmg_direct macro

b34b6c6

feedback: move zend_win_tsrm_cache_init to tsrm.c

bed8bda

feedback: tsrm initializes tls slot

261db20

perf: non-volatile gs read on clang

11afd15

fix tests

6f60a87

fix gdb script

c2d4528

fall back to &_tsrm_ls_cache in case something blew the early TEB slo…

a3c09d3

…t budget

suggestion: TlsFree in case of dll unload or late-TEB fallback

02ae5f7

let ts_free_thread() clear EG(persistent_list) to prevent leaks

6b94d63

set up _tlv_atexit() for macOS plist cleanup

28ea67a

use ir TLS_ADDR

d34ae6b

henderkes force-pushed the perf/1-tls-eg-cg branch from 05d9972 to d34ae6b Compare September 29, 2026 11:21

Merge branch 'master' into perf/1-tls-eg-cg

868c117

arnaud-lb reviewed Oct 12, 2026

View reviewed changes

henderkes mentioned this pull request Oct 13, 2026

fix GH-24262: default apache build to global-dynamic TLS #24301

Open

henderkes added 7 commits October 13, 2026 20:12

macro -> function

961a094

add comment explaining why we use fls instead of just tlsalloc

e28efae

cache index based tls jit too

23f2ab6

figure out linker relaxation for initial-exec

11b9414

fix old typo

1213987

aarch64 too

c83767d

no surplus case gets global dynamic model

865e14d

Conversation

henderkes commented Aug 12, 2026

Uh oh!

arnaud-lb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

henderkes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

henderkes commented Aug 20, 2026

Uh oh!

henderkes commented Aug 20, 2026

Uh oh!

arnaud-lb commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henderkes commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

henderkes commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arnaud-lb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

henderkes commented Sep 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

henderkes commented Sep 12, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

henderkes Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnaud-lb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

henderkes Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

henderkes Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

arnaud-lb commented Aug 20, 2026 •

edited

Loading

henderkes commented Aug 20, 2026 •

edited

Loading

henderkes commented Aug 25, 2026 •

edited

Loading

henderkes Sep 13, 2026 •

edited

Loading

henderkes Oct 15, 2026 •

edited

Loading

henderkes Oct 13, 2026 •

edited

Loading