feat: add before_yield_callback to support post-persistence event filtering #5161

Feature Request

Problem

After #3990 is fixed (PRs #4239 / #5021), on_event_callback will run before append_event, ensuring plugin modifications are persisted. This is the correct fix for the persistence-consistency problem.

However, it eliminates the ability to independently control what gets persisted versus what gets yielded to external consumers. There is currently no plugin callback that runs after session persistence but before the event is yielded.

Use Case

When integrating ADK with external protocols (e.g., AG-UI via ag_ui_adk), it is common to need:

• Full data in the session — the LLM needs complete tool call/response history and all state deltas for correct reasoning across invocations.
• Filtered data in the yield — the external consumer (UI client) should not receive certain internal state fields, or tool call/response events for backend-only tools that have no UI representation.

Examples:

• Stripping internal state keys (e.g., workflow_metadata, internal_cache) from state_delta before they reach the UI, while keeping them in the session for LLM context.
• Suppressing function_call/function_response parts for tools that are purely backend operations (e.g., start_audience_creation, set_audience_name) where the UI only cares about state changes, not the tool invocation itself.

Current Workarounds

Proposed Solution

Add a before_yield_callback (or on_event_yield_callback) to BasePlugin that runs after append_event but before the event is yielded from the runner:

class BasePlugin(ABC):
    # Existing — runs before persistence (after #3990 fix)
    async def on_event_callback(
        self, *, invocation_context: InvocationContext, event: Event
    ) -> Optional[Event]:
        """Modify events before persistence and yielding."""
        pass

    # Proposed — runs after persistence, before yield
    async def before_yield_callback(
        self, *, invocation_context: InvocationContext, event: Event
    ) -> Optional[Event]:
        """Modify or filter the event before it is yielded to external consumers.

        The event has already been persisted to the session at this point.
        Returning a modified Event replaces what is yielded (session unaffected).
        Returning None yields the original persisted event unchanged.
        """
        pass

The runner change in _process_event (from PR #5021) would become:

async def _process_event(event, *, should_append_event):
    # Step 1: on_event_callback (modify for persistence + yield)
    modified_event = await plugin_manager.run_on_event_callback(
        invocation_context=invocation_context, event=event
    )
    final_event = modified_event or event

    # Step 2: Persist
    if should_append_event:
        await self.session_service.append_event(session=session, event=final_event)

    # Step 3: before_yield_callback (modify for yield only)
    yield_event = await plugin_manager.run_before_yield_callback(
        invocation_context=invocation_context, event=final_event
    )
    return yield_event or final_event

Benefits

• Backward compatible — before_yield_callback defaults to pass (returns None), so existing plugins are unaffected.
• Clean separation of concerns — persistence modifications vs. yield filtering are independent.
• Enables middleware integration patterns — ADK + AG-UI, ADK + A2A, or any external protocol consumer can filter events without affecting session integrity.
• Consistent with existing plugin lifecycle — follows the same pattern as before_tool_callback/after_tool_callback split.

Related Issues

• on_event_callback executes after append_event, preventing plugin modifications from being persisted #3990 — on_event_callback executes after append_event, preventing plugin modifications from being persisted
• PR fix: persist events after plugin on_event_callback modifications #4239 — fix: persist events after plugin on_event_callback modifications
• PR fix: persist on_event_callback mutations before session append #5021 — fix: persist on_event_callback mutations before session append