Guide customer-owned safe primary-to-secondary provider fallback

[Hou (SciencePotato)]

Summary

• Document same-channel primary and secondary accounts behind one SAS URL, /api/SendOtp, with customer-selected ordering and no vendor ranking or recommendation.
• Default to no fallback: only policy-approved proven pre-dispatch failure or documented operation-specific nonacceptance may permit one guarded secondary attempt. Timeouts, lost responses, ambiguous errors, blocks and generic status codes do not qualify.
• Explain isolated account credentials, durable operation claims/send intents, aggregate deadlines, lifecycle, existing code seams and separately approved rollout.

Scope

Documentation only: README link and guide. No router, durable store, provider classifier, setup script, test or CI feature ships. The unmodified sample still selects one provider per deployment.

No real provider response is designated safe for fallback; response-based eligibility must remain disabled until provider-specific semantics are reviewed. Existing transport/result helpers do not establish nonacceptance. Store guards do not prove exactly-once delivery or control SAS/native fallback outside the endpoint. Regional Front Door failover remains separate.

Evidence

At reviewed head 9bbf48839e5a6045fd499221578aaabec3d4fb6a, a session-only policy model passed 36 TAP test entries (20 top-level tests, including 16 nested subtests). The coordinator independently reproduced all 36 passes on Node v22.17.1.

The model uses real configuration, delivery, adapter and transport code with fake credentials, HTTP, shared in-memory store and clock. It checks conservative eligibility, concurrent duplicate guards, uncertain-intent retention, budgets, store faults and terminal secondary failure. Provider nonacceptance is an explicit synthetic test attestation, not a validated real provider status. This is not registered-handler integration, distributed-store durability, real deadline/cancellation integration or live delivery proof.

Previous routing/isolation experiments are historical and are not evidence for this fallback design. No Azure, Graph, Key Vault, SAS or provider operations were performed. Static source/link/JSON/JavaScript/fence checks passed; runtime/setup/test/CI paths remain unchanged.

Read the step-by-step guide

Current guide wording reviewed at 95c0251ac778a990cad225d987486da427de5abb: role-based account names and explicit adapter placeholders replace vendor-specific role assignments. Historical model results above retain their original tested revision and fixture scope; this wording-only revision does not add new runtime or live-test evidence.

[Hou (SciencePotato)]

Historical evidence — superseded design: This comment and its harness describe an earlier routing/isolation version of the guide. PR #38 now describes customer-owned primary-to-secondary provider fallback behind one SAS URL. The results below do not validate the new fallback policy. They are preserved for traceability, not as current integration, provider-safety or live-delivery evidence. Current model scope and limitations are in the PR description.

Independent feasibility and accuracy recheck

Latest reviewed revision: 71d2f5d, Node v22.17.1, Windows. Final PR scope remains README + implementation guide only; no runtime, setup script, test or CI changes ship.

Accuracy correction

The previous lazy-acquisition instructions removed the startup function and hook but omitted the bottom module.exports change. The earlier experimental harness supplied that export repair itself. Removing that hidden repair reproduced five ReferenceError: startProviderCredentialRefresh is not defined failures. This was a real gap between the instructions and the earlier experiment, not a production runtime change.

Step 6 now explicitly shows both export variants. The updated harness extracts both from the guide rather than applying an undocumented repair. All 17 registered-handler checks passed in an independent rerun at the revision above. Step 7 also distinguishes running the baseline before edits from adapting singleton/lifecycle fixtures after customization, and supplies a conditional dependency-restore command.

What the evidence supports

The guide's JavaScript customization is feasible with separate per-account settings and credential services. The test applies the guide's context, selection, credential and lifecycle snippets to real SendOtp.js in memory, then invokes its registered HTTP callback and startup/shutdown callbacks. It preserves real parsing, JWE decryption, credential/cache code, adapters, transport, logging, response mapping and catches.

Evidence	Result
Latest guide-derived registered-handler checks	17 passed, 0 failed
Existing adapter/cache/handler checks, previous reviewed revision 72fcd28	35 passed, 0 failed; runtime source unchanged since that run
Original context-isolation experiment, previous reviewed revision 72fcd28	6 passed, 0 failed; historical supplementary evidence
SMS/Telesign, voice/Soprano and same-provider accounts	Routing, credentials, wire shaping, nonce/correlation and isolation checked
Evaluation, failures, simulated refresh/expiry and shutdown	Ordering, no fallback and cleanup checked
Negative control: reuse one service across accounts/auth modes	Retains first bundle; separate services remain necessary

Limits: Functions registration, SDKs, provider HTTP and refresh scheduling are faked. This is not an Azure Functions host/Easy Auth, real SAS, OAuth, Key Vault, entitlement or handset test. The synthetic customer loader is not a production schema. Distributed workers, real scheduling/reload, trusted customer authorization and .NET/Python multi-context execution remain outside the evidence. No Azure deployment, secret reads or live sends were needed for this review.

Latest independently rerun handler checks

ok 1 - registered handler executes guide SMS/Telesign and voice/Soprano with exact wire responses
ok 2 - cold concurrent requests coalesce independently through the registered handler
ok 3 - two accounts of the same adapter never share credentials under concurrent SMS/voice traffic
ok 4 - evaluation exits before context lookup or any credential/HTTP call, even with no usable route
ok 5 - warm lifecycle evaluation adds no request-driven acquisition or provider transport
ok 6 - unknown route and channel mismatch fail closed before any credential/HTTP use
ok 7 - registered response mapping preserves block, throttle, rejection and malformed-response failures without fallback
ok 8 - real transport timeout maps to 504 and does not select an alternate context
ok 9 - credential failures stop only their request and never borrow another context
ok 10 - registered startup reports token failure while other context remains usable
ok 11 - scheduled refresh publishes isolated new credentials and shutdown removes all refresh schedules
ok 12 - expiry and failed refresh fail closed while an independent OAuth context keeps working
ok 13 - termination aborts in-flight acquisition and late fake results cannot repopulate a stopped cache
ok 14 - header correlation fallback and body precedence remain unchanged on the actual wire
# BOUNDARY RESULT: zero external network attempts; zero application/credential environment reads or writes; all real singleton caches unused.
ok 15 - caller-supplied provider selectors cannot override the configured channel route
ok 16 - customer startup validation rejects duplicate contexts and inconsistent routes before registration or SDK use
ok 17 - negative control retains the first bundle if one credential service is reused across accounts or modes
# tests 17
# pass 17
# fail 0
# cancelled 0
# skipped 0
# todo 0

Reproduce the offline evidence (not a supported shipped command)

Use a disposable checkout of commit 71d2f5d91d4a7d88ae91b09bf67a7baf93d9cfec with JavaScript dependencies restored from the lockfile. Save the harness below outside the repository, then run from the repository root:

node --test "<absolute-path-to-registered-handler-feasibility.cjs>"

The harness reads actual source and guide snippets from the current directory and fails on unexpected drift. It writes no repository files. Keep all fakes and network/configuration guards in place; do not substitute real credentials or services.
Harness SHA-256 (UTF-8, LF newlines): dec7d8529bbcb3d73596c5ea8bbca4533d2005b682ee74accb23ad36b5351915

'use strict';

// Session-only harness. Run from the repository root; no repository files are written.
// It extracts guide snippets, edits the actual SendOtp source in memory, then invokes
// the callback registered by app.http and the actual registered lifecycle callbacks.
const { test, mock, after } = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const Module = require('node:module');
const crypto = require('node:crypto');
const { execFileSync } = require('node:child_process');
const root = process.cwd();
const functionDir = path.join(root, 'javascript', 'src', 'functions');
const handlerPath = path.join(functionDir, 'SendOtp.js');
const credentialsPath = path.join(functionDir, 'credentials.js');
const guidePath = path.join(root, 'docs', 'MULTI-PROVIDER-IMPLEMENTATION.md');
const source = fs.readFileSync(handlerPath, 'utf8').replace(/\r\n/g, '\n');
const credentialSource = fs.readFileSync(credentialsPath, 'utf8');
const guide = fs.readFileSync(guidePath, 'utf8').replace(/\r\n/g, '\n');
const hash = (value) => crypto.createHash('sha256').update(value).digest('hex');
const fromApp = Module.createRequire(path.join(root, 'javascript', 'package.json'));
const jsBlocks = [...guide.matchAll(/```javascript\n([\s\S]*?)\n```/g)].map(x => x[1]);
function snippet(marker) {
    const found = jsBlocks.filter(x => x.includes(marker));
    assert.equal(found.length, 1, `Expected one guide snippet containing ${marker}`);
    return found[0];
}
function replaceOnce(text, old, replacement) {
    assert.equal(text.split(old).length, 2, `Source drift: ${old.slice(0, 100)}`);
    return text.replace(old, replacement);
}

// Direct application of steps 3-6; parsing, JWE, delivery, response and error code stay intact.
let adapted = replaceOnce(source, '    credentialTokenService,\n', '    CredentialTokenService,\n');
const start = adapted.indexOf('async function startProviderCredentialRefresh()');
const end = adapted.indexOf("app.hook.appStart(startProviderCredentialRefresh);");
assert.ok(start > 0 && end > start);
const contextCode = snippet('const entries = CUSTOMER_LOAD_AND_VALIDATE_CONFIG()')
    .split('\n').filter(line => !/^const .*require\(/.test(line)).join('\n');
const routeCode = snippet('const routeByChannel = Object.freeze(');
const lifecycleCode = snippet('async function startProviderCredentialRefresh()');
const prewarmExports = snippet('module.exports = { startProviderCredentialRefresh, stopProviderCredentialRefresh };');
const lazyExports = snippet('module.exports = { stopProviderCredentialRefresh };');
assert.ok(source.includes(prewarmExports));
adapted = adapted.slice(0, start) + routeCode + '\n' + contextCode + '\n' + lifecycleCode + '\n\n' + adapted.slice(end);
const selectionStart = adapted.indexOf('            const provider = selectProvider(config.providerName);');
const selectionEnd = adapted.indexOf('            logContext = providerContext(logContext, provider);', selectionStart);
assert.ok(selectionStart > 0 && selectionEnd > selectionStart);
adapted = adapted.slice(0, selectionStart)
    + snippet('const contextId = routeByChannel[payload.channelName];').split('\n').map(x => '            ' + x).join('\n')
    + '\n' + adapted.slice(selectionEnd);
// Only outbound references change. Assert expected counts to reject drift.
for (const [old, next, count] of [
    ['config.providerChannel', 'providerConfig.providerChannel', 2],
    ['config.providerAuthMode', 'providerConfig.providerAuthMode', 2],
    ['config.providerEndpoint', 'providerConfig.providerEndpoint', 2],
    ['config.providerTimeoutMs', 'providerConfig.providerTimeoutMs', 1],
    ['config.env', 'providerConfig.env', 1],
]) {
    // Scope replacements to the registered handler, not initialization or selectedContext.config.
    const marker = adapted.indexOf("app.http('SendOtp'");
    const tail = adapted.slice(marker);
    const pattern = new RegExp(`(?<![\\w.])${old.replace('.', '\\.')}`, 'g');
    assert.equal([...tail.matchAll(pattern)].length, count, `Unexpected reference count: ${old}`);
    adapted = adapted.slice(0, marker) + tail.replace(pattern, next);
}
adapted = replaceOnce(adapted, 'credentialContext(logContext, config)', 'credentialContext(logContext, providerConfig)');
adapted = replaceOnce(adapted,
    'credential = await credentialTokenService.getCredentials(\n                    provider.credentialSpec,\n                    config,\n                );',
    snippet('credential = await credentials.getCredentials('));
assert.ok(adapted.includes('config.decryptionKeyPem') && adapted.includes('config.expectedKeyId'));
assert.ok(!adapted.includes('credentialTokenService'));
assert.equal(adapted.slice(adapted.indexOf('app.http')).split('const config = readConfig();').length, 2);
// Test instrumentation only: expose contexts without altering their implementation.
adapted += '\nmodule.exports.testContexts = contexts;\n';

const metadata = {
    command: `node --test "${__filename}"`,
    cwd: root,
    node: process.version,
    sourceCommit: execFileSync('git', ['rev-parse', 'HEAD'], { encoding: 'utf8' }).trim(),
    handlerSha256: hash(fs.readFileSync(handlerPath)),
    credentialsSha256: hash(fs.readFileSync(credentialsPath)),
    guideSha256: hash(fs.readFileSync(guidePath)),
    experimentSha256: hash(fs.readFileSync(__filename)),
    adaptedHandlerSha256: hash(adapted),
    boundaries: 'Fake Functions registration, Azure SDK, fetch and refresh clock/scheduler; real registered handler and credential/cache logic.',
};
console.log('EVIDENCE ' + JSON.stringify(metadata));

// Do not snapshot/read real environment values. Install a synthetic, read-only facade
// once for the test process; the handler receives its inbound settings explicitly.
const nativeEnv = process.env;
const nodeTestContext = nativeEnv.NODE_TEST_CONTEXT; // Node's non-secret test-runner IPC marker.
let environmentViolations = 0;
const denyEnvironmentWrite = () => {
    environmentViolations++;
    throw new Error('Environment mutation forbidden');
};
const guardedEnv = new Proxy(Object.create(null), {
    get(_target, key) {
        if (key === 'NODE_TEST_CONTEXT') return nodeTestContext;
        if (typeof key === 'string' && /^(EPP_|AZURE_|KEY_VAULT|GH_|GITHUB_|COPILOT_)/.test(key)) {
            environmentViolations++;
            throw new Error('Real application/credential environment access forbidden');
        }
        return undefined;
    },
    set: denyEnvironmentWrite,
    deleteProperty: denyEnvironmentWrite,
    defineProperty: denyEnvironmentWrite,
});
process.env = guardedEnv;

let networkAttempts = 0;
const forbidden = () => { networkAttempts++; throw new Error('External access forbidden'); };
for (const [object, name] of [
    [require('node:net').Socket.prototype, 'connect'],
    [require('node:net'), 'connect'], [require('node:net'), 'createConnection'],
    [require('node:tls'), 'connect'],
    [require('node:http'), 'request'], [require('node:http'), 'get'],
    [require('node:https'), 'request'], [require('node:https'), 'get'],
    [require('node:dns'), 'lookup'], [require('node:dns').promises, 'lookup'],
    [globalThis, 'fetch'],
]) mock.method(object, name, forbidden);

const { readConfig } = fromApp('./src/functions/config');
const { selectProvider } = fromApp('./src/functions/providers');
const { isValidProviderUrl } = fromApp('./src/functions/providerTransport');
const { CompactEncrypt } = fromApp('jose');
const { publicKey, privateKey } = crypto.generateKeyPairSync('rsa', { modulusLength: 2048 });
const pem = privateKey.export({ format: 'pem', type: 'pkcs8' });
const flush = () => new Promise(setImmediate);
const message = 'SYNTHETIC message 001234.';
function compile(filename, code, dependencies, prefix = '') {
    const mod = new Module(filename, module);
    mod.filename = filename;
    mod.paths = Module._nodeModulePaths(path.dirname(filename));
    const realRequire = Module.createRequire(filename);
    mod.require = (name) => {
        if (Object.hasOwn(dependencies, name)) return dependencies[name];
        if (name.startsWith('@azure/')) throw new Error('Unexpected real Azure SDK import');
        return realRequire(name);
    };
    mod._compile(prefix + code, filename);
    return mod.exports;
}
function entry(id, provider, channel) {
    return { id, settings: {
        EPP_PROVIDER_NAME: provider, EPP_PROVIDER_CHANNEL: channel,
        EPP_PROVIDER_AUTH_MODE: provider === 'telesign' ? 'apiKey' : 'oauth',
        EPP_PROVIDER_ENDPOINT: `https://${id}.invalid/messages`,
        EPP_PROVIDER_TIMEOUT_MS: '1500',
        KEY_VAULT_URL: `https://${id}.invalid`, AZURE_CLIENT_ID: `mi-${id}`,
        EPP_PROVIDER_TENANT_ID: `tenant-${id}`, EPP_PROVIDER_SCOPE: `api://${id}/.default`,
        EPP_OUTBOUND_CLIENT_ID: `app-${id}`, EPP_OUTBOUND_MI_CLIENT_ID: `mi-${id}`,
    } };
}
const defaultEntries = () => [
    entry('telesign-primary', 'telesign', 'sms'),
    entry('soprano-primary', 'soprano', 'voice'),
];

async function envelope(id, channel = 1, overrides = {}, delivery = {}) {
    const encryptedDeliveryContext = await new CompactEncrypt(Buffer.from(JSON.stringify({
        nonce: `SYNTHETIC-NONCE-${id}`, phoneNumber: '+15551234567',
        message, locale: 'fr-FR', ...delivery,
    }))).setProtectedHeader({ alg: 'RSA-OAEP-256', enc: 'A256GCM', kid: 'local-inbound-key' }).encrypt(publicKey);
    return {
        type: 'microsoft.mfa.otpDeliver.v1', channel, mode: 1, ttlSeconds: 60,
        correlationId: id, encryptedDeliveryContext, ...overrides,
    };
}

function fixture(t, options = {}) {
    const entries = options.entries || defaultEntries();
    const routing = options.routing || { sms: 'telesign-primary', voice: 'soprano-primary' };
    const h = {
        sdk: [], services: [], credentialCalls: [], failures: [], requests: [], logs: [], violations: [],
        timers: new Set(), failSecrets: new Set(), failTokens: new Set(), responses: new Map(),
        now: Date.now(), version: 1, gate: Promise.resolve(), hookCounts: { start: 0, stop: 0 },
    };
    function check(action) {
        try { return action(); } catch (error) { h.violations.push(error.message); throw error; }
    }
    class FakeIdentity {
        constructor(settings) { this.id = settings.clientId; }
        async getToken(scope) {
            h.sdk.push({ kind: 'identity', id: this.id, scope });
            check(() => assert.equal(scope, 'api://AzureADTokenExchange/.default'));
            await h.gate;
            return { token: `SYNTHETIC-ASSERTION-${this.id}`, expiresOnTimestamp: h.now + 3600000 };
        }
    }
    class FakeAssertionCredential {
        constructor(tenant, client, assertion) { Object.assign(this, { tenant, client, assertion }); }
        async getToken(scope) {
            const id = this.client.slice('app-'.length);
            h.sdk.push({ kind: 'oauth', id, scope });
            check(() => {
                assert.equal(this.tenant, `tenant-${id}`);
                assert.equal(scope, `api://${id}/.default`);
            });
            await h.gate;
            check(() => assert.ok(entries.some(x => x.id === id)));
            if (h.failTokens.has(id)) throw new Error('SYNTHETIC-TOKEN-FAILURE');
            const assertion = await this.assertion();
            check(() => assert.equal(assertion, `SYNTHETIC-ASSERTION-mi-${id}`));
            return { token: `SYNTHETIC-TOKEN-${id}-${h.version}`, expiresOnTimestamp: h.now + 3600000 };
        }
    }
    class FakeSecretClient {
        constructor(url, identity) { Object.assign(this, { url, identity }); }
        async getSecret(name) {
            const id = new URL(this.url).hostname.replace('.invalid', '');
            h.sdk.push({ kind: 'secret', id, name });
            check(() => {
                assert.ok(entries.some(x => x.id === id));
                assert.equal(this.identity.id, `mi-${id}`);
                assert.ok(['telesign-api-key', 'telesign-customer-id'].includes(name));
            });
            await h.gate;
            if (h.failSecrets.has(id)) throw new Error('SYNTHETIC-SECRET-FAILURE');
            return { value: name === 'telesign-api-key'
                ? `SYNTHETIC-KEY-${id}-${h.version}` : `SYNTHETIC-ACCOUNT-${id}-${h.version}` };
        }
    }
    const actual = compile(credentialsPath, credentialSource, {
        '@azure/identity': { ManagedIdentityCredential: FakeIdentity,
            ClientAssertionCredential: FakeAssertionCredential,
            AzureAuthorityHosts: { AzurePublicCloud: 'https://identity.invalid' } },
        '@azure/keyvault-secrets': { SecretClient: FakeSecretClient },
        '@azure/core-rest-pipeline': { createDefaultHttpClient: () => ({ sendRequest: forbidden }) },
        '@azure/logger': { AzureLogger: { log: forbidden } },
    });
    class InstrumentedService extends actual.CredentialTokenService {
        constructor() {
            super({
                cacheOptions: {
                    now: () => h.now,
                    schedule: (callback, delay) => {
                        check(() => assert.equal(delay, 30000));
                        const timer = { callback, unref() {} };
                        h.timers.add(timer);
                        return timer;
                    },
                    cancel: (timer) => h.timers.delete(timer),
                },
                reportFailure: (kind) => h.failures.push(kind),
            });
            h.services.push(this);
        }
        getCredentials(spec, config) {
            h.credentialCalls.push({ service: this, spec, config });
            return super.getCredentials(spec, config);
        }
    }
    const inbound = Object.freeze({
        EPP_DECRYPTION_KEY_PEM: pem, EPP_ENCRYPTION_KEY_ID: 'local-inbound-key',
        // Deliberately contradictory outbound process settings must never drive selection.
        EPP_PROVIDER_NAME: 'unknown-global-provider',
        EPP_PROVIDER_ENDPOINT: 'https://wrong-context.invalid',
    });
    const hooks = {
        appStart(fn) { h.hookCounts.start++; h.start = fn; },
        appTerminate(fn) { h.hookCounts.stop++; h.stop = fn; },
    };
    const registrations = [];
    const dependencies = {
        '@azure/functions': { app: { hook: hooks, http(name, registration) {
            registrations.push({ name, registration }); h.handler = registration.handler;
        } } },
        './config': { readConfig: (env) => readConfig(env === undefined ? inbound : env) },
        './credentials': { CredentialTokenService: InstrumentedService,
            reportRefreshFailure: (kind) => h.failures.push(kind) },
        'test-only-customer-loader': () => {
            const ids = new Set();
            // This is the CUSTOMER_* implementation for synthetic fixtures, not a shipped schema.
            for (const e of entries) {
                if (!e.id || ids.has(e.id)) throw new Error('Missing or duplicate provider context ID');
                ids.add(e.id);
                const c = readConfig(e.settings), p = selectProvider(c.providerName);
                if (!p || p.authenticationMode !== c.providerAuthMode
                    || !['sms', 'voice'].includes(c.providerChannel)
                    || !isValidProviderUrl(c.providerEndpoint)
                    || c.providerEndpoint !== `https://${e.id}.invalid/messages`
                    || !/^\d+$/.test(c.providerTimeoutMs)) throw new Error('Invalid synthetic configuration');
            }
            for (const [channel, id] of Object.entries(routing)) {
                const match = entries.find(x => x.id === id);
                if (!match || match.settings.EPP_PROVIDER_CHANNEL !== channel) {
                    throw new Error('Invalid synthetic routing rule');
                }
            }
            return entries;
        },
    };
    let code = adapted;
    if (options.routing) code = replaceOnce(code, routeCode,
        `const routeByChannel = Object.freeze(${JSON.stringify(routing)});`);
    if (options.lazy) {
        code = replaceOnce(code, lifecycleCode,
            lifecycleCode.slice(lifecycleCode.indexOf('function stopProviderCredentialRefresh()')));
        code = replaceOnce(code, 'app.hook.appStart(startProviderCredentialRefresh);', '');
        code = replaceOnce(code, prewarmExports, lazyExports);
    }
    h.exports = compile(handlerPath, code, dependencies,
        "const CUSTOMER_LOAD_AND_VALIDATE_CONFIG = require('test-only-customer-loader');\n");
    h.contexts = h.exports.testContexts;
    assert.equal(registrations.length, 1);
    assert.equal(registrations[0].name, 'SendOtp');
    assert.deepEqual(registrations[0].registration.methods, ['POST']);
    assert.equal(registrations[0].registration.authLevel, 'anonymous'); // Easy Auth is outside this unit harness.
    assert.deepEqual(h.hookCounts, { start: options.lazy ? 0 : 1, stop: 1 });
    for (const context of h.contexts.values()) {
        assert.ok(Object.isFrozen(context.config) && Object.isFrozen(context.config.env));
        assert.equal(context.config.decryptionKeyPem, '');
    }
    t.mock.method(globalThis, 'fetch', async (url, args) => {
        const id = new URL(url).hostname.replace('.invalid', '');
        const e = entries.find(x => x.id === id);
        check(() => {
            assert.ok(e, 'Unexpected endpoint');
            assert.equal(url, e.settings.EPP_PROVIDER_ENDPOINT);
            assert.equal(args.method, 'POST');
            assert.equal(args.redirect, 'manual');
            assert.ok(args.signal instanceof AbortSignal);
            const expected = e.settings.EPP_PROVIDER_NAME === 'telesign'
                ? `Basic ${Buffer.from(`SYNTHETIC-ACCOUNT-${id}-${h.version}:SYNTHETIC-KEY-${id}-${h.version}`).toString('base64')}`
                : `Bearer SYNTHETIC-TOKEN-${id}-${h.version}`;
            assert.equal(args.headers.Authorization, expected);
            assert.notEqual(args.headers.Authorization, 'SYNTHETIC-INBOUND-AUTH');
        });
        const body = JSON.parse(args.body);
        const correlation = body.correlation_id || body.correlationId;
        h.requests.push({ id, body, correlation });
        const mode = h.responses.get(id);
        if (mode === 'network') throw new Error('SYNTHETIC-NETWORK-FAILURE');
        if (mode === 'timeout') {
            return new Promise((resolve, reject) => args.signal.addEventListener('abort',
                () => reject(new Error('SYNTHETIC-TIMEOUT')), { once: true }));
        }
        await flush();
        const defaultBody = e.settings.EPP_PROVIDER_NAME === 'telesign'
            ? { status: { code: 290 }, reference_id: correlation }
            : { status: 'ENROUTE', id: correlation };
        const status = mode?.status || 200;
        return { status, ok: status >= 200 && status < 300,
            text: async () => mode?.text ?? JSON.stringify(mode?.body ?? defaultBody) };
    });
    h.invoke = async (body, headers = {}) => {
        const records = [];
        const capture = (value) => { const record = JSON.parse(value); records.push(record); h.logs.push(record); };
        const response = await h.handler({
            text: async () => typeof body === 'string' ? body : JSON.stringify(body),
            headers: { get: (name) => headers[name] ?? null },
        }, { invocationId: 'synthetic-invocation', log: capture, warn: capture, error: capture });
        assert.equal(records.filter(x => x.eventName === 'request_completed').length, 1);
        return { ...response, records };
    };
    h.advance = async (milliseconds) => {
        h.now += milliseconds;
        for (const timer of h.timers) timer.callback();
        await Promise.allSettled(h.services.map(s => s.pending).filter(Boolean));
    };
    t.after(() => {
        h.stop();
        assert.equal(h.timers.size, 0);
        assert.ok(h.services.every(x => x.closed && (!x.current || !x.current.get())));
        assert.equal(actual.credentialTokenService.current, null, 'Exported singleton must remain unused');
        assert.equal(networkAttempts, 0);
        assert.deepEqual(h.violations, [], 'Assertions must not be swallowed by handler error mapping');
        assert.doesNotMatch(JSON.stringify(h.logs), /SYNTHETIC-(KEY|ACCOUNT|TOKEN|ASSERTION|NONCE|INBOUND-AUTH|SECRET-FAILURE|NETWORK-FAILURE)|SYNTHETIC message|15551234567/);
        assert.strictEqual(process.env, guardedEnv);
        assert.equal(environmentViolations, 0, 'No environment reads/writes for application configuration');
    });
    return h;
}
function success(response, id) {
    assert.equal(response.status, 200);
    assert.deepEqual(response.jsonBody, {
        nonce: `SYNTHETIC-NONCE-${id}`, correlationId: id, providerStatus: 'accepted',
    });
}
function failure(response, status, reason) {
    assert.equal(response.status, status);
    assert.equal(response.jsonBody.error, 'provider_delivery_failed');
    assert.equal(typeof response.jsonBody.requestId, 'string');
    assert.ok(!Object.hasOwn(response.jsonBody, 'nonce'));
    assert.ok(response.records.some(x => x.eventName === 'request_failed' && x.failureReason === reason));
}

test('registered handler executes guide SMS/Telesign and voice/Soprano with exact wire responses', async (t) => {
    const h = fixture(t);
    await h.start();
    const input = await Promise.all(Array.from({ length: 24 }, (_, i) => envelope(`wire-${i}`, i % 2 ? 2 : 1)));
    const results = await Promise.all(input.map(x => h.invoke(x, {
        authorization: 'SYNTHETIC-INBOUND-AUTH', 'x-ms-correlation-id': 'header-ignored',
    })));
    results.forEach((x, i) => success(x, `wire-${i}`));
    assert.equal(h.requests.length, 24);
    for (const request of h.requests) {
        const number = Number(request.correlation.slice('wire-'.length));
        assert.equal(request.id, number % 2 ? 'soprano-primary' : 'telesign-primary');
        if (number % 2) {
            assert.equal(request.body.destination, '15551234567');
            assert.deepEqual(request.body.messageTypes, ['voice']);
            assert.deepEqual(request.body.voice.text2voice, {
                beforePasswordText: 'SYNTHETIC message ', password: '001234',
                afterPasswordText: '.', language: 'fr-FR', gender: 1, loop: 2,
            });
        } else {
            assert.equal(request.body.recipient.phone_number, '+15551234567');
            assert.equal(request.body.message.text, message);
            assert.equal(request.body.message.language, 'fr-FR');
            assert.deepEqual(request.body.channels, [{ channel: 'sms' }]);
        }
    }
    assert.equal(h.sdk.filter(x => x.kind === 'secret').length, 2);
    assert.equal(h.sdk.filter(x => x.kind === 'oauth').length, 1);
    assert.equal(h.timers.size, 2);
    assert.ok(results[1].records.some(x => x.providerTenantId === 'tenant-soprano-primary'));
    assert.ok(results.every(x => !x.records.some(r => r.eventName === 'encryption_key_id_mismatch')));
});

test('cold concurrent requests coalesce independently through the registered handler', async (t) => {
    const h = fixture(t, { lazy: true });
    let release;
    h.gate = new Promise(resolve => { release = resolve; });
    const bodies = await Promise.all(Array.from({ length: 20 }, (_, i) => envelope(`cold-${i}`, i % 2 ? 2 : 1)));
    const pending = bodies.map(x => h.invoke(x));
    try {
        for (let i = 0; i < 200 && h.sdk.length < 3; i++) await flush();
        assert.equal(h.sdk.filter(x => x.kind === 'secret').length, 2);
        assert.equal(h.sdk.filter(x => x.kind === 'identity').length, 1);
        assert.ok(h.services.every(x => x.pending));
    } finally { release(); }
    const results = await Promise.all(pending);
    results.forEach((x, i) => success(x, `cold-${i}`));
    assert.equal(h.sdk.filter(x => x.kind === 'oauth').length, 1);
    assert.equal(h.requests.length, 20);
});

test('two accounts of the same adapter never share credentials under concurrent SMS/voice traffic', async (t) => {
    const entries = [entry('telesign-primary', 'telesign', 'sms'), entry('telesign-secondary', 'telesign', 'voice')];
    const h = fixture(t, { entries, routing: { sms: 'telesign-primary', voice: 'telesign-secondary' } });
    await h.start();
    const bodies = await Promise.all(Array.from({ length: 16 }, (_, i) => envelope(`accounts-${i}`, i % 2 ? 2 : 1)));
    const results = await Promise.all(bodies.map(x => h.invoke(x)));
    results.forEach((x, i) => success(x, `accounts-${i}`));
    assert.equal(h.sdk.filter(x => x.kind === 'secret').length, 4);
    assert.equal(h.sdk.filter(x => x.kind === 'oauth').length, 0);
    assert.notEqual(h.services[0].current, h.services[1].current);
    for (const request of h.requests) {
        const isVoice = Number(request.correlation.slice('accounts-'.length)) % 2;
        assert.equal(request.id, isVoice ? 'telesign-secondary' : 'telesign-primary');
        assert.equal(request.body.message.text, isVoice
            ? 'SYNTHETIC message 0, 0, 1, 2, 3, 4. SYNTHETIC message 0, 0, 1, 2, 3, 4.'
            : message);
    }
});

test('evaluation exits before context lookup or any credential/HTTP call, even with no usable route', async (t) => {
    const h = fixture(t, { lazy: true });
    // Make every route inaccessible without removing service ownership needed by the stop hook.
    const lookup = t.mock.method(h.contexts, 'get', () => { throw new Error('Route lookup forbidden for evaluation'); });
    success(await h.invoke(await envelope('eval', 2, { mode: 2 })), 'eval');
    const malformed = await h.invoke(await envelope('bad-eval', 1, { mode: 2 }, { nonce: '' }));
    assert.equal(malformed.status, 400);
    assert.equal(malformed.jsonBody.reason, 'incomplete delivery context');
    const corrupt = await h.invoke(await envelope('corrupt', 1, { mode: 2, encryptedDeliveryContext: 'invalid-jwe' }));
    assert.equal(corrupt.status, 400);
    assert.equal(corrupt.jsonBody.error, 'decryption_failed');
    assert.equal(lookup.mock.callCount(), 0);
    assert.equal(h.credentialCalls.length, 0);
    assert.equal(h.sdk.length, 0);
    assert.equal(h.requests.length, 0);
    assert.equal(h.timers.size, 0);
});

test('warm lifecycle evaluation adds no request-driven acquisition or provider transport', async (t) => {
    const h = fixture(t);
    await h.start();
    const sdkCount = h.sdk.length, calls = h.credentialCalls.length;
    success(await h.invoke(await envelope('warm-eval', 1, { mode: 2 })), 'warm-eval');
    assert.equal(h.sdk.length, sdkCount);
    assert.equal(h.credentialCalls.length, calls);
    assert.equal(h.requests.length, 0);
});

test('unknown route and channel mismatch fail closed before any credential/HTTP use', async (t) => {
    const h = fixture(t, { lazy: true });
    const soprano = h.contexts.get('soprano-primary');
    h.contexts.delete('soprano-primary');
    failure(await h.invoke(await envelope('unknown', 2)), 400, 'unknown_provider');
    h.contexts.set('soprano-primary', Object.freeze({
        ...soprano, config: Object.freeze({ ...soprano.config, providerChannel: 'sms' }),
    }));
    failure(await h.invoke(await envelope('mismatch', 2)), 400, 'channel_not_configured');
    assert.equal(h.sdk.length, 0);
    assert.equal(h.requests.length, 0);
});

test('registered response mapping preserves block, throttle, rejection and malformed-response failures without fallback', async (t) => {
    const h = fixture(t);
    await h.start();
    const cases = [
        [{ body: { status: 'BLOCKED' } }, 403, 'provider_rejected'],
        [{ status: 429, body: { status: 'FAILED' } }, 429, 'provider_http_error'],
        [{ status: 500 }, 502, 'provider_http_error'],
        [{ text: 'SYNTHETIC-PRIVATE-NONJSON' }, 502, 'invalid_provider_json'],
        [{ body: { status: 'UNKNOWN-PRIVATE-STATUS' } }, 502, 'unrecognized_provider_status'],
        ['network', 502, 'provider_network_error'],
    ];
    for (let i = 0; i < cases.length; i++) {
        const [response, status, reason] = cases[i];
        h.responses.set('soprano-primary', response);
        const results = await Promise.all([
            h.invoke(await envelope(`failure-${i}`, 2)),
            h.invoke(await envelope(`healthy-${i}`, 1)),
        ]);
        failure(results[0], status, reason);
        success(results[1], `healthy-${i}`);
        assert.equal(h.requests.filter(x => x.correlation === `failure-${i}`).length, 1);
        assert.equal(h.requests.find(x => x.correlation === `failure-${i}`).id, 'soprano-primary');
    }
    assert.equal(h.requests.length, 12);
    assert.doesNotMatch(JSON.stringify(h.logs), /UNKNOWN-PRIVATE-STATUS|SYNTHETIC-PRIVATE-NONJSON/);
});

test('real transport timeout maps to 504 and does not select an alternate context', async (t) => {
    const entries = defaultEntries();
    entries[1].settings.EPP_PROVIDER_TIMEOUT_MS = '5';
    const h = fixture(t, { entries });
    h.responses.set('soprano-primary', 'timeout');
    failure(await h.invoke(await envelope('timeout', 2)), 504, 'provider_timeout');
    assert.deepEqual(h.requests.map(x => x.id), ['soprano-primary']);
});

test('credential failures stop only their request and never borrow another context', async (t) => {
    const h = fixture(t, { lazy: true });
    h.failSecrets.add('telesign-primary');
    const results = await Promise.all([
        h.invoke(await envelope('bad-credentials', 1)),
        h.invoke(await envelope('other-credentials', 2)),
    ]);
    failure(results[0], 502, 'credential_unavailable');
    success(results[1], 'other-credentials');
    assert.deepEqual(h.requests.map(x => x.id), ['soprano-primary']);
    assert.ok(h.failures.includes('key_vault'));
});

test('registered startup reports token failure while other context remains usable', async (t) => {
    const h = fixture(t);
    h.failTokens.add('soprano-primary');
    await h.start();
    assert.ok(h.failures.includes('provider_token'));
    failure(await h.invoke(await envelope('bad-token', 2)), 502, 'credential_unavailable');
    success(await h.invoke(await envelope('good-key', 1)), 'good-key');
    assert.deepEqual(h.requests.map(x => x.id), ['telesign-primary']);
});

test('scheduled refresh publishes isolated new credentials and shutdown removes all refresh schedules', async (t) => {
    const h = fixture(t);
    await h.start();
    h.version = 2;
    await h.advance(240000);
    success(await h.invoke(await envelope('refreshed-key', 1)), 'refreshed-key');
    success(await h.invoke(await envelope('refreshed-token', 2)), 'refreshed-token');
    assert.equal(h.sdk.filter(x => x.kind === 'secret').length, 4);
    assert.equal(h.sdk.filter(x => x.kind === 'oauth').length, 2);
    h.stop();
    assert.equal(h.timers.size, 0);
    const count = h.sdk.length;
    await h.advance(300000);
    assert.equal(h.sdk.length, count);
    failure(await h.invoke(await envelope('after-stop', 1)), 502, 'credential_unavailable');
    assert.equal(h.requests.length, 2);
});

test('expiry and failed refresh fail closed while an independent OAuth context keeps working', async (t) => {
    const h = fixture(t);
    await h.start();
    h.failSecrets.add('telesign-primary');
    await h.advance(300001);
    failure(await h.invoke(await envelope('expired-key', 1)), 502, 'credential_unavailable');
    success(await h.invoke(await envelope('healthy-token', 2)), 'healthy-token');
    assert.deepEqual(h.requests.map(x => x.id), ['soprano-primary']);
});

test('termination aborts in-flight acquisition and late fake results cannot repopulate a stopped cache', async (t) => {
    const h = fixture(t, { lazy: true });
    let release;
    h.gate = new Promise(resolve => { release = resolve; });
    const pending = h.invoke(await envelope('shutdown-pending', 1));
    try {
        for (let i = 0; i < 200 && h.sdk.length < 2; i++) await flush();
        assert.equal(h.sdk.length, 2);
        h.stop();
        failure(await pending, 502, 'credential_unavailable');
    } finally { release(); }
    await flush();
    assert.equal(h.requests.length, 0);
    assert.ok(h.services.every(x => !x.current || x.current.get() === null));
});

test('header correlation fallback and body precedence remain unchanged on the actual wire', async (t) => {
    const h = fixture(t);
    const first = await h.invoke(await envelope('body-wins'), { 'x-ms-correlation-id': 'header-loses' });
    success(first, 'body-wins');
    const second = await h.invoke(await envelope('header', 1, { correlationId: undefined }),
        { 'x-ms-correlation-id': 'header-wins' });
    assert.equal(second.status, 200);
    assert.equal(second.jsonBody.correlationId, 'header-wins');
    assert.equal(second.jsonBody.nonce, 'SYNTHETIC-NONCE-header');
    assert.deepEqual(h.requests.map(x => x.correlation), ['body-wins', 'header-wins']);
});

test('caller-supplied provider selectors cannot override the configured channel route', async (t) => {
    const h = fixture(t);
    const response = await h.invoke(await envelope('untrusted-selector', 1, {
        providerId: 'soprano-primary', tenantId: 'untrusted-customer',
        endpoint: 'https://caller-controlled.invalid',
    }), {
        'x-provider-id': 'soprano-primary', 'x-provider-endpoint': 'https://caller-controlled.invalid',
    });
    success(response, 'untrusted-selector');
    assert.deepEqual(h.requests.map(x => x.id), ['telesign-primary']);
    assert.equal(h.sdk.filter(x => x.kind === 'oauth').length, 0);
});

test('customer startup validation rejects duplicate contexts and inconsistent routes before registration or SDK use', (t) => {
    const entries = defaultEntries();
    assert.throws(() => fixture(t, { entries: [entries[0], entries[0]] }), /duplicate/);
    assert.throws(() => fixture(t, { routing: { sms: 'missing', voice: 'soprano-primary' } }),
        /Invalid synthetic routing/);
    assert.throws(() => fixture(t, { routing: { sms: 'soprano-primary', voice: 'telesign-primary' } }),
        /Invalid synthetic routing/);
});

test('negative control retains the first bundle if one credential service is reused across accounts or modes', async (t) => {
    const h = fixture(t);
    const a = h.contexts.get('telesign-primary'), b = h.contexts.get('soprano-primary');
    const first = await a.credentials.getCredentials(a.provider.credentialSpec, a.config);
    const other = readConfig(entry('other-telesign', 'telesign', 'sms').settings);
    const wrongAccount = await a.credentials.getCredentials(a.provider.credentialSpec, other);
    const wrongMode = await a.credentials.getCredentials(b.provider.credentialSpec, b.config);
    assert.strictEqual(first, wrongAccount);
    assert.strictEqual(first, wrongMode);
    assert.equal(wrongMode.mode, 'apiKey');
    assert.equal(h.sdk.filter(x => x.kind === 'secret').length, 2);
    assert.equal(h.sdk.filter(x => x.kind === 'oauth').length, 0);
    assert.equal(h.requests.length, 0); // Never dispatch with the deliberately misbound bundle.
});

after(() => {
    assert.equal(networkAttempts, 0);
    assert.equal(environmentViolations, 0);
    assert.strictEqual(process.env, guardedEnv);
    process.env = nativeEnv;
    mock.restoreAll();
    console.log('BOUNDARY RESULT: zero external network attempts; zero application/credential environment reads or writes; all real singleton caches unused.');
});