Skip to content

inlineData image parts dropped when using /run endpoint #4975

Closed
Bug
Closed
inlineData image parts dropped when using /run endpoint#4975
Bug
Assignees
sanketpatil06
Labels
models[Component] This issue is related to model supportrequest clarification[Status] The maintainer need clarification or more information from the author

Description

@envyram
envyram
opened on Mar 24, 2026

Bug Report: inlineData image parts dropped when using /run endpoint

Description

When sending a multimodal message (text + image) to the /run endpoint, the inlineData part is not forwarded to the underlying LLM. The image arrives at the ADK server correctly but is silently dropped before the LiteLLM call, resulting in the model receiving an empty images: [] array.

Environment

  • ADK version: latest
  • LiteLLM version: 1.82.4
  • Model: ollama_chat/ministral-3:8b via Ollama
  • Ollama: local instance at localhost:11434

Steps to Reproduce

  1. Start an ADK agent with a vision-capable Ollama model:
root_agent = Agent(
    model=LiteLlm(model="ollama_chat/ministral-3:8b"),
    name="assistant_agent",
    ...
)
  1. Send a multimodal request to /run with both text and image parts:
{
  "appName": "default",
  "userId": "user",
  "sessionId": "test",
  "newMessage": {
    "role": "user",
    "parts": [
      {"text": "What is in this image?"},
      {
        "inlineData": {
          "mimeType": "image/png",
          "data": "<base64-encoded-image>"
        }
      }
    ]
  },
  "streaming": false
}
  1. Observe the model responds as if no image was provided.

Debug Evidence

With

import litellm
litellm._turn_on_debug()

it can be seen that the outgoing request to Ollama shows the image array is empty despite being present in the /run payload:

POST http://localhost:11434/api/chat
{
  "messages": [
    {"role": "user", "content": "What is in this image?", "images": []}
  ]
}

The inlineData base64 bytes are never populated into the images field.

Expected Behavior

The inlineData part from the /run payload should be converted and forwarded as base64 image data in the outgoing LLM request.

Actual Behavior

The images field is sent as an empty array []. The model responds as if no image was attached.

Additional Notes

The same model (ministral-3:8b) correctly processes images when called directly via the Ollama native UI or via the Ollama Python SDK, confirming the model itself supports vision. The issue is specific to ADK's message conversion layer between the /run API and the LiteLLM call.

Activity

  1. adk-bot
    added
    models[Component] This issue is related to model support
    on Mar 24, 2026
  2. adk-bot
    added the
    Bug
    issue type on Mar 24, 2026
  3. klateefa
    assigned
    klateefa
    and unassigned
    klateefa
    on Mar 25, 2026

  4. weiguangli-io commented on Mar 25, 2026

    @weiguangli-io
    weiguangli-io
    on Mar 25, 2026
    Contributor

    Root Cause

    The bug is in _flatten_ollama_content() at src/google/adk/models/lite_llm.py (lines 1154-1192), called by _normalize_ollama_chat_messages() (line 1195).

    When the model string starts with ollama_chat, generate_content_async() normalizes the messages via _normalize_ollama_chat_messages() (line 2198). This calls _flatten_ollama_content(), which iterates the multipart content array and keeps only blocks with type == "text" (line 1181-1184), silently dropping all image_url / video_url / audio_url blocks.

    So a well-formed multipart message:

    [
    {"type": "text", "text": "What is in this image?"},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
]

    gets flattened to:

    "What is in this image?"

    LiteLLM then receives a plain string (no multipart array), so it has nothing to populate Ollama's images field with → images: [].

    Proposed Fix

    Modify _flatten_ollama_content() to return the original multipart content list when it contains non-text media blocks (image_url, video_url, audio_url). LiteLLM's Ollama handler already knows how to convert image_url content blocks into Ollama's native images array, so the flattening should only apply when the content is truly text-only:

    _MEDIA_BLOCK_TYPES = {"image_url", "video_url", "audio_url"}

def _flatten_ollama_content(
    content: OpenAIMessageContent | str | None,
) -> OpenAIMessageContent | str | None:
    if content is None or isinstance(content, str):
        return content

    if isinstance(content, dict):
        try:
            return json.dumps(content)
        except TypeError:
            return str(content)

    try:
        blocks = list(content)
    except TypeError:
        return str(content)

    # If any block is a media type, keep the full multipart array
    # so LiteLLM can convert it to Ollama's native format.
    has_media = any(
        isinstance(b, dict) and b.get("type") in _MEDIA_BLOCK_TYPES
        for b in blocks
    )
    if has_media:
        return blocks

    # Text-only: flatten to a single string for compatibility.
    text_parts = []
    for block in blocks:
        if isinstance(block, dict) and block.get("type") == "text":
            text_value = block.get("text")
            if text_value:
                text_parts.append(text_value)

    if text_parts:
        return _NEW_LINE.join(text_parts)

    try:
        return json.dumps(blocks)
    except TypeError:
        return str(blocks)

    Happy to submit a PR if this approach looks good.

  5. envyram commented on Mar 25, 2026

    @envyram
    envyram
    on Mar 25, 2026
    Author

    Massively surprised no one has hit this issue so far - this is kind of a blocker.

  6. envyram commented on Mar 30, 2026

    @envyram
    envyram
    on Mar 30, 2026
    Author

    Hi - following up on this please

  7. saivedant169
    added a commit that references this issue on Apr 12, 2026

    fix(models): preserve media blocks in _flatten_ollama_content

    059e1b3

  8. sanketpatil06 commented on Apr 23, 2026

    @sanketpatil06
    sanketpatil06
    on Apr 23, 2026

    Hi @envyram , Thank you for reporting this. A Pull Request (#5296) has been submitted to address this issue and is currently being reviewed. While we review it, would you mind testing it on your side to confirm if it fixes the problem? Thanks

  9. surajksharma07
    assigned
    sanketpatil06
    on Apr 23, 2026
  10. surajksharma07
    added
    request clarification[Status] The maintainer need clarification or more information from the author
    on Apr 23, 2026
  11. saivedant169
    added a commit that references this issue on May 17, 2026

    fix(models): preserve media blocks in _flatten_ollama_content

    e4be567

  12. sanketpatil06 commented on May 22, 2026

    @sanketpatil06
    sanketpatil06
    on May 22, 2026

    Hi @envyram , We are closing this issue because it has been inactive for while without the requested details. If you still need help, please feel free to reach out.
    Thanks.

  13. sanketpatil06
    closed this as completedon May 22, 2026
  14. copybara-service
    added 2 commits that reference this issue on Jun 2, 2026

    fix: preserve media blocks in ollama content flattening

    47ceeba

    fix: preserve media blocks in ollama content flattening

    63ce7d3
  15. FrigaZzz
    added a commit that references this issue on Aug 11, 2026

    fix: preserve media blocks in ollama content flattening

    8e7ab1b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

models[Component] This issue is related to model supportrequest clarification[Status] The maintainer need clarification or more information from the author

Type

Bug

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions