feat: support multimodal input in agent chat

[andreavs]

Summary

• Add ImageContent message content type with from_file() and from_bytes() helpers for loading images from disk
• Extend Message to accept multimodal content as a list of text and image parts
• Update agent response loading to handle content-parts arrays

Associated service contract change: https://github.com/cognitedata/service-contracts/pull/3303

Manual Test

To see the new feature in action, try out the following script:

#!/usr/bin/env python3
"""Manual test script for multimodal agent chat with image input.

Downloads a sample image, creates a temporary agent, and asks what is in the image.

Prerequisites:
    - Authenticated CogniteClient environment (e.g. ``cognite login`` or env vars).
    - A CDF project with Atlas AI / Agents enabled.
    - A vision-capable language model (override with the ``AGENT_MODEL`` env var).

Usage:
    python scripts/test_multimodal_agent_chat.py
"""

from __future__ import annotations

import os
import sys
import tempfile
import uuid
from pathlib import Path
from urllib.request import urlretrieve

from cognite.client import ClientConfig, CogniteClient
from cognite.client.credentials import OAuthClientCredentials
from cognite.client.data_classes.agents import AgentUpsert, ImageContent, Message, TextContent

IMAGE_URL = "https://cave.cs.toronto.edu/kriz/cifar-10-sample/cat1.png"
DEFAULT_MODEL = "azure/gpt-5.4"
VARS_FILE = Path(__file__).resolve().parent.parent / "vars.txt"


def main() -> int:
    client = CogniteClient()
    agent_external_id = f"sdk_multimodal_test_agent_{uuid.uuid4().hex[:12]}"
    image_path: Path | None = None

    try:
        with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as tmp:
            image_path = Path(tmp.name)

        print(f"Downloading image from {IMAGE_URL} ...")
        urlretrieve(IMAGE_URL, image_path)
        print(f"Saved image to {image_path}")

        model = os.environ.get("AGENT_MODEL", DEFAULT_MODEL)
        print(f"Creating agent {agent_external_id!r} (model={model!r}) ...")
        agent = client.agents.upsert(
            AgentUpsert(
                external_id=agent_external_id,
                name="Multimodal test agent",
                description="Temporary agent for testing image input",
                instructions="You are a helpful assistant that can analyze images.",
                model=model,
            )
        )
        print(f"Created agent: {agent.name}")

        print('Sending chat message: "What is in this image?"')
        response = client.agents.chat(
            agent_external_id=agent.external_id,
            messages=Message(
                [
                    TextContent("What is in this image?"),
                    ImageContent.from_file(image_path),
                ]
            ),
        )

        print("\n--- Response ---")
        print(f"type: {response.type}")
        print(f"cursor: {response.cursor}")
        if response.text:
            print(f"text: {response.text}")
        else:
            print("messages:")
            for index, message in enumerate(response.messages or []):
                print(f"  [{index}] role={message.role} content={message.content!r}")

        return 0
    finally:
        if image_path is not None and image_path.exists():
            image_path.unlink()
            print(f"\nRemoved temporary image {image_path}")

        print(f"Deleting agent {agent_external_id!r} ...")
        try:
            client.agents.delete(external_ids=agent_external_id, ignore_unknown_ids=True)
            print("Agent deleted.")
        except Exception as exc:
            print(f"Warning: could not delete agent: {exc}", file=sys.stderr)


if __name__ == "__main__":
    raise SystemExit(main())

Test plan

• Unit tests for ImageContent.from_file / from_bytes
• Unit tests for multimodal Message serialization
• Unit tests for multimodal chat request body
• Unit tests for loading agent responses with content-parts arrays

Made with Cursor

[codecov]

Codecov Report

❌ Patch coverage is 97.82609% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.66%. Comparing base (7d5e3df) to head (3b31601).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
cognite/client/data_classes/agents/chat.py	95.38%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2717      +/-   ##
==========================================
- Coverage   93.70%   93.66%   -0.05%     
==========================================
  Files         504      504              
  Lines       51120    51248     +128     
==========================================
+ Hits        47904    48003      +99     
- Misses       3216     3245      +29     

Files with missing lines	Coverage Δ
cognite/client/data_classes/agents/__init__.py	100.00% <ø> (ø)
tests/tests_unit/test_api/test_agents_chat.py	100.00% <100.00%> (ø)
cognite/client/data_classes/agents/chat.py	97.09% <95.38%> (-0.43%)	⬇️

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gemini-code-assist]

Code Review

This pull request introduces support for multimodal messages in the agents chat module by adding the ImageContent class and updating Message and AgentMessage to handle sequences of content parts. Feedback recommends checking the raw byte size of image data before base64 encoding to prevent memory overhead, and validating sequence elements in Message.__init__ to ensure type safety and prevent runtime crashes.

[polomani]

LGTM

[polomani]

shouldn't ToolConfirmationCall._load_call support the array now as well?

[andreavs]

If I understand the question correctly, then no, because the tool confirmation call is made by the agent to the user - we now support multimodal input, but not multimodal output

[polomani]

update from master so CI succeeds

[nithinb]

risk review ok