Skip to content

feat(fetch-url): support fetching images as multimodal content #1100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bj456736 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat(fetch-url): support fetching images as multimodal content #1100

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ed6d2a1
feat(fetch-url): support fetching images as multimodal content
Jun 25, 2026
a97fad4
feat(fetch-url): align with internal API shape
Jun 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/agent-core/src/tools/builtin/web/fetch-url.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
Fetch content from a URL. Returns the main text content extracted from the page. Use this when you need to read a specific web page.
Fetch content from a URL. Returns the main text content extracted from the page, or the image data if the URL points to an image file. Use this when you need to read a specific web page or image.

Only public `http`/`https` URLs are supported. Requests to private, loopback, or link-local addresses are refused, and responses larger than 10 MiB are rejected.
34 changes: 26 additions & 8 deletions packages/agent-core/src/tools/builtin/web/fetch-url.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

import type { BuiltinTool } from '../../../agent/tool';
import { ToolAccesses } from '../../../loop/tool-access';
import type { ExecutableToolContext, ExecutableToolResult, ToolExecution } from '../../../loop/types';
import type { ContentPart, ExecutableToolContext, ExecutableToolResult, ToolExecution } from '../../../loop/types';

Check failure on line 13 in packages/agent-core/src/tools/builtin/web/fetch-url.ts

View workflow job for this annotation

GitHub Actions / typecheck 

Module '"../../../loop/types"' declares 'ContentPart' locally, but it is not exported.
import { toInputJsonSchema } from '../../support/input-schema';
import { literalRulePattern, matchesGlobRuleSubject } from '../../support/rule-match';
import { ToolResultBuilder } from '../../support/result-builder';
Expand All @@ -25,14 +25,18 @@
* returned verbatim, in full.
* - `extracted` — the body was an HTML page; only the main article text
* was extracted and returned.
* - `image` — the body is an image file; the binary data is returned
* as base64-encoded content for multimodal model input.
*/
export type UrlFetchKind = 'passthrough' | 'extracted';
export type UrlFetchKind = 'passthrough' | 'extracted' | 'image';

export interface UrlFetchResult {
/** The text handed to the LLM. */
/** The text handed to the LLM, or empty string for image content. */
content: string;
/** Whether `content` is a verbatim passthrough or extracted main text. */
/** Whether content is a verbatim passthrough, extracted main text, or image data. */
kind: UrlFetchKind;
/** Image data as base64, when kind is 'image'. */
image?: { mimeType: string; base64: string };
}

export interface UrlFetcher {
Expand Down Expand Up @@ -84,12 +88,26 @@

private async execution(
args: FetchURLInput,
{
toolCallId,
}: ExecutableToolContext,
{ toolCallId }: ExecutableToolContext,
): Promise<ExecutableToolResult> {
try {
const { content, kind } = await this.fetcher.fetch(args.url, { toolCallId });
const { content, kind, image } = await this.fetcher.fetch(args.url, { toolCallId });

if (image) {
const output: ContentPart[] = [
{
type: 'text',
text: `<system>Fetched image from ${args.url}. Mime type: ${image.mimeType}</system>`,
},
{ type: 'text', text: `<image url="${args.url}">` },
{
type: 'image_url',
imageUrl: { url: `data:${image.mimeType};base64,${image.base64}` },

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 25, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Gate fetched images on image-capable models

When the active model lacks image_in, FetchURL is still registered whenever a URL fetcher exists (checked packages/agent-core/src/agent/tool/index.ts, where only ReadMediaFile is capability-gated). Fetching any image/* URL now returns an image_url part, which providers serialize as image input on the next request, so text-only aliases can fail after a successful fetch instead of receiving an actionable tool error. Pass model capabilities into this tool or degrade/error before emitting the image_url part.

Useful? React with 👍 / 👎.

},
{ type: 'text', text: '</image>' },
];
return { output, isError: false };
}

if (!content) {
return {
Expand Down
23 changes: 21 additions & 2 deletions packages/agent-core/src/tools/providers/local-fetch-url.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
* 3. Reject responses larger than `maxBytes` (content-length first,
* then measured body length as a defensive second check).
* 4. `text/plain` / `text/markdown` → passthrough verbatim.
* 5. Otherwise (assumed HTML) → run Readability over a linkedom
* 5. `image/*` → download binary, encode as base64, return as image kind.
* 6. Otherwise (assumed HTML) → run Readability over a linkedom
* document. Return `# ${title}\n\n${text}` (title omitted when
* absent). If extraction yields no meaningful text, fall back to
* common content containers (`<article>` / `<main>` / `<body>`)
Expand Down Expand Up @@ -172,6 +173,25 @@ export class LocalFetchURLProvider implements UrlFetcher {
}
}

const contentType = (response.headers.get('content-type') ?? '').toLowerCase();

// Handle image content types
if (contentType.startsWith('image/')) {
const arrayBuffer = await response.arrayBuffer();
const actualBytes = arrayBuffer.byteLength;
if (actualBytes > this.maxBytes) {
throw new Error(
`Response body too large: ${String(actualBytes)} bytes exceeds maxBytes (${String(this.maxBytes)}).`,
);
}
const base64 = Buffer.from(arrayBuffer).toString('base64');
return {
content: '',
kind: 'image',
image: { mimeType: contentType, base64 },

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 25, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restrict fetched images to supported MIME types

This accepts every image/* content type and stores the full header as the MIME type. For URLs returning image/svg+xml, image/avif, or a header with parameters such as image/png; charset=binary, FetchURLTool emits a data: image URL that our Anthropic converter rejects because it only allows image/png, image/jpeg, image/gif, and image/webp (packages/kosong/src/providers/anthropic.ts:418-430). Restrict or sanitize the MIME type before returning kind: 'image', otherwise a successful fetch can make the following model request fail.

Useful? React with 👍 / 👎.

};
}

const body = await response.text();

// Servers may omit content-length — measure again defensively.
Expand All @@ -182,7 +202,6 @@ export class LocalFetchURLProvider implements UrlFetcher {
);
}

const contentType = (response.headers.get('content-type') ?? '').toLowerCase();
if (contentType.startsWith('text/plain') || contentType.startsWith('text/markdown')) {
return { content: body, kind: 'passthrough' };
}
Expand Down
22 changes: 22 additions & 0 deletions packages/agent-core/test/tools/providers/local-fetch-url.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,29 @@ function htmlResponse(body: string, contentType: string): Response {
});
}

function imageResponse(data: Uint8Array, contentType: string): Response {
return new Response(data, {
status: 200,
headers: { 'content-type': contentType },
});
}

describe('LocalFetchURLProvider content kind', () => {
it('reports image content as image kind with base64 data', async () => {
const imageData = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
const fetchImpl = vi
.fn<typeof fetch>()
.mockResolvedValue(imageResponse(imageData, 'image/png'));
const provider = new LocalFetchURLProvider({ fetchImpl });

const result = await provider.fetch('https://example.com/image.png');

expect(result.kind).toBe('image');
expect(result.image).toBeDefined();
expect(result.image?.mimeType).toBe('image/png');
expect(result.image?.base64).toBeTruthy();
});

it('reports text/plain bodies as a verbatim passthrough', async () => {
const fetchImpl = vi
.fn<typeof fetch>()
Expand Down
Loading