diff --git a/Makefile b/Makefile index 6ab2fba96e..9d57cd2bca 100644 --- a/Makefile +++ b/Makefile @@ -32,10 +32,10 @@ check-uv-version: build: check-uv-version @echo "$(CYAN)Setting up OpenHands V1 development environment...$(RESET)" @echo "$(YELLOW)Installing dependencies with uv sync --dev...$(RESET)" - @uv sync --dev + @env -u VIRTUAL_ENV uv sync --dev @echo "$(GREEN)Dependencies installed successfully.$(RESET)" @echo "$(YELLOW)Setting up pre-commit hooks...$(RESET)" - @uv run pre-commit install + @env -u VIRTUAL_ENV uv run pre-commit install @echo "$(GREEN)Pre-commit hooks installed successfully.$(RESET)" @echo "$(GREEN)Build complete! Development environment is ready.$(RESET)" diff --git a/docs/responses-api.md b/docs/responses-api.md new file mode 100644 index 0000000000..91f6459ddc --- /dev/null +++ b/docs/responses-api.md @@ -0,0 +1,93 @@ +Title: OpenAI Responses API via LiteLLM (gpt-5) — Inputs, Tools, Reasoning, Stateful vs Stateless + +Overview +- This doc explains how the OpenAI Responses API works when called natively through LiteLLM in this project. +- You’ll find concrete JSON examples under examples/responses_api/ and guidance on stateless/stateful usage, images, tool calls, and reasoning (including encrypted reasoning). + +Install and build +1) make build + - Uses uv to create a local .venv and install dependencies (including litellm and openai) + - Installs pre-commit hooks + +Key packages present in .venv +- litellm (responses support lives under litellm/responses/) +- openai (official types for Responses API) + +How LiteLLM wires to the OpenAI Responses API +- Entrypoints: litellm/responses/main.py + - responses(...) and aresponses(...) implement the OpenAI Responses API surface. + - If the provider supports native Responses, LiteLLM uses BaseLLMHTTPHandler.response_api_handler to call it. + - If a provider lacks native Responses, LiteLLM transforms to a compatible flow via litellm.responses.litellm_completion_transformation. +- Utilities: litellm/responses/utils.py + - ResponsesAPIRequestUtils.get_optional_params_responses_api(...) maps/filters optional params per provider. + - convert_text_format_to_text_param(...) converts a Pydantic/dict schema to the text.format payload. + - _build/_decode_responses_api_response_id(...) encodes provider+model into response ids; previous_response_id is decoded before sending upstream. + - ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage maps Responses usage to common Usage. +- MCP integration: aresponses_api_with_mcp(...) in main.py handles MCP tools, auto-exec if configured, and merges events into streamed output. + +Core OpenAI Responses API types (from openai-python) +- Input items: response_input_item.py and response_input_content.py + - Message: { type: "message", role: "user"|"system"|"developer", content: [ ... ] } + - Content list items include: + - input_text { type: "input_text", text } + - input_image { type: "input_image", image_url | file_id, detail } + - input_file, input_audio +- Tools: tool.py + - Variants include function, file_search, computer, web_search, mcp, code_interpreter, image_generation, local_shell, custom_tool (and preview variants) + - Function tool calls are ResponseFunctionToolCall objects; follow-ups use function_call_output items +- Reasoning: shared/reasoning.py and responses/response_reasoning_item.py + - request.reasoning: { effort: minimal|low|medium|high, summary: auto|concise|detailed } + - response contains reasoning items: { type: "reasoning", summary: [...], content?: [...], encrypted_content?: string } + - To receive encrypted reasoning text, include: ["reasoning.encrypted_content"] + +Stateless vs. stateful +- Stateless: You send full context in input each call. No previous_response_id. +- Stateful: Set store: true and reuse previous_response_id from the prior response. LiteLLM encodes provider+model into ids; it decodes before hitting OpenAI upstream but you pass the encoded id back in subsequent calls. + +Examples in this repo (JSON) +1) System + User with images (stateless) + - examples/responses_api/01_system_user_images.json +2) Tool call (function); then follow-up with tool output + - examples/responses_api/02_tool_call_and_output.json + - examples/responses_api/02b_tool_call_followup_with_output_item.json + - Two tools, model returns two tool calls + a reasoning item (stateless, no encrypted reasoning): examples/responses_api/02c_two_tool_calls_with_reasoning.json + - Follow-up sending both tool outputs back, including a reasoning item: examples/responses_api/02d_two_tool_call_outputs_followup_with_reasoning_item.json +3) Reasoning (stateless) + - Plain reasoning (no encrypted content): examples/responses_api/03_stateless_reasoning_plain.json + - Encrypted reasoning included: examples/responses_api/03b_stateless_reasoning_encrypted.json + - Follow-up including the returned reasoning item (plain): examples/responses_api/03c_stateless_reasoning_plain_followup.json + - Follow-up including the returned reasoning item (encrypted): examples/responses_api/03d_stateless_reasoning_encrypted_followup.json +4) Reasoning (stateful) + - Plain: examples/responses_api/04_stateful_reasoning_plain.json + - Encrypted reasoning included: examples/responses_api/04b_stateful_reasoning_encrypted.json + - Follow-up including the returned reasoning item (plain): examples/responses_api/04c_stateful_reasoning_plain_followup.json + - Follow-up including the returned reasoning item (encrypted): examples/responses_api/04d_stateful_reasoning_encrypted_followup.json +5) Reasoning response snapshots (shape returned by API) + - examples/responses_api/05_reasoning_response_examples.json + +Notes on correctness +- Content discriminator fields must match the OpenAI types (e.g., input_text, input_image). +- For function call follow-up, use type: "function_call_output" with the prior call_id and an output that MUST be a JSON string (per OpenAI types). +- To request encrypted reasoning, add "reasoning.encrypted_content" to include. The API then returns reasoning items with encrypted_content. +- previous_response_id should be the id returned from the prior Responses call. When using LiteLLM, that id includes encoded provider/model info; return it as-is in the next call. + +Minimal Python usage sketch +""" +from openai import OpenAI + +client = OpenAI() +resp = client.responses.create( + model="gpt-5", + input=[ + {"type": "message", "role": "user", "content": [ + {"type": "input_text", "text": "Hello"} + ]}, + ], + reasoning={"effort": "low", "summary": "concise"}, + include=["reasoning.encrypted_content"], +) +print(resp.id, resp.output_text) +""" + +Troubleshooting +- VIRTUAL_ENV mismatch: use uv run commands (e.g., uv run pytest, uv run python) or activate the project .venv explicitly. LiteLLM/OpenAI should import from /agent-sdk/.venv. diff --git a/examples/responses_api/01_system_user_images.json b/examples/responses_api/01_system_user_images.json new file mode 100644 index 0000000000..af8a65cd6b --- /dev/null +++ b/examples/responses_api/01_system_user_images.json @@ -0,0 +1,30 @@ +{ + "model": "gpt-5", + "instructions": "You are a precise assistant.", + "input": [ + { + "type": "message", + "role": "system", + "content": [ + { "type": "input_text", "text": "Answer concisely in under 20 words." } + ] + }, + { + "type": "message", + "role": "user", + "content": [ + { "type": "input_text", "text": "What's in these images?" }, + { + "type": "input_image", + "detail": "auto", + "image_url": "https://upload.wikimedia.org/wikipedia/commons/9/99/Black_square.jpg" + }, + { + "type": "input_image", + "detail": "high", + "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAAEklEQVR42mP8/5+hHgMDAwMABRYC2r8v8bEAAAAASUVORK5CYII=" + } + ] + } + ] +} diff --git a/examples/responses_api/02_tool_call_and_output.json b/examples/responses_api/02_tool_call_and_output.json new file mode 100644 index 0000000000..a94425fb3d --- /dev/null +++ b/examples/responses_api/02_tool_call_and_output.json @@ -0,0 +1,34 @@ +{ + "model": "gpt-5", + "instructions": "You are a helpful assistant.", + "tools": [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the weather for a city", + "parameters": { + "type": "object", + "properties": { + "city": {"type": "string"}, + "unit": {"type": "string", "enum": ["c", "f"]} + }, + "required": ["city"] + }, + "strict": true + } + } + ], + "input": [ + { + "type": "message", + "role": "user", + "content": [ + {"type": "input_text", "text": "What's the weather in Paris in Celsius?"} + ] + } + ], + "include": ["message.output_text.logprobs"], + "max_output_tokens": 300, + "tool_choice": {"type": "function", "function": {"name": "get_weather"}} +} diff --git a/examples/responses_api/02b_tool_call_followup_with_output_item.json b/examples/responses_api/02b_tool_call_followup_with_output_item.json new file mode 100644 index 0000000000..4728513b30 --- /dev/null +++ b/examples/responses_api/02b_tool_call_followup_with_output_item.json @@ -0,0 +1,11 @@ +{ + "model": "gpt-5", + "previous_response_id": "resp_abc123", + "input": [ + { + "type": "function_call_output", + "call_id": "call_12345", + "output": "{\"temperature\":15,\"unit\":\"c\",\"description\":\"sunny\"}" + } + ] +} diff --git a/examples/responses_api/02c_two_tool_calls_with_reasoning.json b/examples/responses_api/02c_two_tool_calls_with_reasoning.json new file mode 100644 index 0000000000..a5a7d0028e --- /dev/null +++ b/examples/responses_api/02c_two_tool_calls_with_reasoning.json @@ -0,0 +1,48 @@ +{ + "model": "gpt-5", + "instructions": "Use the available tools. Return a reasoning item as needed.", + "tools": [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get weather for a city", + "parameters": { + "type": "object", + "properties": { + "city": {"type": "string"}, + "unit": {"type": "string", "enum": ["c", "f"]} + }, + "required": ["city"] + }, + "strict": true + } + }, + { + "type": "function", + "function": { + "name": "get_local_time", + "description": "Get local time for a city", + "parameters": { + "type": "object", + "properties": { + "city": {"type": "string"} + }, + "required": ["city"] + }, + "strict": true + } + } + ], + "input": [ + { + "type": "message", + "role": "user", + "content": [ + {"type": "input_text", "text": "Is it a good time to go for a walk in Paris?"} + ] + } + ], + "reasoning": {"effort": "medium", "summary": "concise"}, + "tool_choice": "auto" +} diff --git a/examples/responses_api/02d_two_tool_call_outputs_followup_with_reasoning_item.json b/examples/responses_api/02d_two_tool_call_outputs_followup_with_reasoning_item.json new file mode 100644 index 0000000000..bc7d01b62e --- /dev/null +++ b/examples/responses_api/02d_two_tool_call_outputs_followup_with_reasoning_item.json @@ -0,0 +1,26 @@ +{ + "model": "gpt-5", + "previous_response_id": "resp_two_tools_prev_id", + "input": [ + { + "type": "function_call_output", + "call_id": "call_weather_1", + "output": "{\"temperature\":18,\"unit\":\"c\",\"description\":\"partly cloudy\"}" + }, + { + "type": "function_call_output", + "call_id": "call_local_time_1", + "output": "{\"local_time\":\"2025-10-03T15:45:00+02:00\"}" + }, + { + "type": "reasoning", + "id": "rsn_tools_01", + "summary": [ + {"type": "summary_text", "text": "Combined weather and local time to decide suitability."} + ], + "content": [ + {"type": "reasoning_text", "text": "18C and partly cloudy at mid-afternoon; generally favorable for a walk."} + ] + } + ] +} diff --git a/examples/responses_api/03_stateless_reasoning_plain.json b/examples/responses_api/03_stateless_reasoning_plain.json new file mode 100644 index 0000000000..73518b2371 --- /dev/null +++ b/examples/responses_api/03_stateless_reasoning_plain.json @@ -0,0 +1,18 @@ +{ + "model": "gpt-5", + "instructions": "Reason step by step, but only include the final answer in output text.", + "input": [ + { + "type": "message", + "role": "user", + "content": [ + {"type": "input_text", "text": "Prove that the sum of two even numbers is even."} + ] + } + ], + "include": [], + "reasoning": { + "effort": "medium", + "summary": "concise" + } +} diff --git a/examples/responses_api/03b_stateless_reasoning_encrypted.json b/examples/responses_api/03b_stateless_reasoning_encrypted.json new file mode 100644 index 0000000000..ecdfa2dc88 --- /dev/null +++ b/examples/responses_api/03b_stateless_reasoning_encrypted.json @@ -0,0 +1,18 @@ +{ + "model": "gpt-5", + "instructions": "Use hidden chain-of-thought; do not reveal intermediate steps unless included.", + "input": [ + { + "type": "message", + "role": "user", + "content": [ + {"type": "input_text", "text": "Compute 1234 * 5678 and explain briefly."} + ] + } + ], + "include": ["reasoning.encrypted_content"], + "reasoning": { + "effort": "high", + "summary": "detailed" + } +} diff --git a/examples/responses_api/03c_stateless_reasoning_plain_followup.json b/examples/responses_api/03c_stateless_reasoning_plain_followup.json new file mode 100644 index 0000000000..7db9443875 --- /dev/null +++ b/examples/responses_api/03c_stateless_reasoning_plain_followup.json @@ -0,0 +1,23 @@ +{ + "model": "gpt-5", + "previous_response_id": "resp_rsn_plain_prev_id", + "input": [ + { + "type": "reasoning", + "id": "rsn_001", + "summary": [ + { "type": "summary_text", "text": "Outlined that even + even = even by factoring 2." } + ], + "content": [ + { "type": "reasoning_text", "text": "Let even numbers be 2a and 2b. Then 2a+2b=2(a+b), which is even." } + ] + }, + { + "type": "message", + "role": "user", + "content": [ + { "type": "input_text", "text": "Using your previous reasoning, generalize to sum of two multiples of 3." } + ] + } + ] +} diff --git a/examples/responses_api/03d_stateless_reasoning_encrypted_followup.json b/examples/responses_api/03d_stateless_reasoning_encrypted_followup.json new file mode 100644 index 0000000000..5896a6c8d8 --- /dev/null +++ b/examples/responses_api/03d_stateless_reasoning_encrypted_followup.json @@ -0,0 +1,22 @@ +{ + "model": "gpt-5", + "previous_response_id": "resp_rsn_encrypted_prev_id", + "include": ["reasoning.encrypted_content"], + "input": [ + { + "type": "reasoning", + "id": "rsn_100", + "summary": [ + { "type": "summary_text", "text": "Computed product and formed concise explanation." } + ], + "encrypted_content": "A1B2C3D4E5F6...encrypted bytes..." + }, + { + "type": "message", + "role": "user", + "content": [ + { "type": "input_text", "text": "Please square 4321 now, using the same approach." } + ] + } + ] +} diff --git a/examples/responses_api/04_stateful_reasoning_plain.json b/examples/responses_api/04_stateful_reasoning_plain.json new file mode 100644 index 0000000000..a6e9a24b82 --- /dev/null +++ b/examples/responses_api/04_stateful_reasoning_plain.json @@ -0,0 +1,16 @@ +{ + "model": "gpt-5", + "store": true, + "instructions": "Continue the conversation from the previous response.", + "previous_response_id": "resp_litellm_encoded_prev_id", + "input": [ + { + "type": "message", + "role": "user", + "content": [ + {"type": "input_text", "text": "Why is the sky blue?"} + ] + } + ], + "reasoning": {"effort": "minimal", "summary": "concise"} +} diff --git a/examples/responses_api/04b_stateful_reasoning_encrypted.json b/examples/responses_api/04b_stateful_reasoning_encrypted.json new file mode 100644 index 0000000000..07ef8f3750 --- /dev/null +++ b/examples/responses_api/04b_stateful_reasoning_encrypted.json @@ -0,0 +1,17 @@ +{ + "model": "gpt-5", + "store": true, + "instructions": "Continue with hidden reasoning included for debugging.", + "previous_response_id": "resp_litellm_encoded_prev_id", + "include": ["reasoning.encrypted_content"], + "input": [ + { + "type": "message", + "role": "user", + "content": [ + {"type": "input_text", "text": "Summarize our last discussion in one sentence."} + ] + } + ], + "reasoning": {"effort": "medium", "summary": "concise"} +} diff --git a/examples/responses_api/04c_stateful_reasoning_plain_followup.json b/examples/responses_api/04c_stateful_reasoning_plain_followup.json new file mode 100644 index 0000000000..d91f6d4a1f --- /dev/null +++ b/examples/responses_api/04c_stateful_reasoning_plain_followup.json @@ -0,0 +1,24 @@ +{ + "model": "gpt-5", + "store": true, + "previous_response_id": "resp_litellm_encoded_prev_id", + "input": [ + { + "type": "reasoning", + "id": "rsn_stateful_plain_01", + "summary": [ + { "type": "summary_text", "text": "Outlined Rayleigh scattering mechanism briefly." } + ], + "content": [ + { "type": "reasoning_text", "text": "Short wavelength light scatters more; blue dominates sky color." } + ] + }, + { + "type": "message", + "role": "user", + "content": [ + { "type": "input_text", "text": "Extend the explanation to twilight colors." } + ] + } + ] +} diff --git a/examples/responses_api/04d_stateful_reasoning_encrypted_followup.json b/examples/responses_api/04d_stateful_reasoning_encrypted_followup.json new file mode 100644 index 0000000000..27c954eff9 --- /dev/null +++ b/examples/responses_api/04d_stateful_reasoning_encrypted_followup.json @@ -0,0 +1,23 @@ +{ + "model": "gpt-5", + "store": true, + "previous_response_id": "resp_litellm_encoded_prev_id", + "include": ["reasoning.encrypted_content"], + "input": [ + { + "type": "reasoning", + "id": "rsn_stateful_enc_01", + "summary": [ + { "type": "summary_text", "text": "Summarized prior discussion into one sentence." } + ], + "encrypted_content": "ZZYYXX...encrypted bytes..." + }, + { + "type": "message", + "role": "user", + "content": [ + { "type": "input_text", "text": "Continue with a 2-sentence summary including key causes and effects." } + ] + } + ] +} diff --git a/examples/responses_api/05_reasoning_response_examples.json b/examples/responses_api/05_reasoning_response_examples.json new file mode 100644 index 0000000000..17df8b0df0 --- /dev/null +++ b/examples/responses_api/05_reasoning_response_examples.json @@ -0,0 +1,65 @@ +{ + "stateless_plain": { + "id": "resp_stateless_plain_001", + "object": "response", + "created_at": 1738531200, + "model": "gpt-5", + "status": "completed", + "output": [ + { + "type": "reasoning", + "id": "rsn_001", + "status": "completed", + "summary": [ + { "type": "summary_text", "text": "Outlined that even + even = even by factoring 2." } + ], + "content": [ + { "type": "reasoning_text", "text": "Let even numbers be 2a and 2b. Then 2a+2b=2(a+b), which is even." } + ] + }, + { + "type": "message", + "id": "msg_001", + "status": "completed", + "role": "assistant", + "content": [ + { "type": "output_text", "text": "Because 2a+2b = 2(a+b), the sum is even." } + ] + } + ], + "parallel_tool_calls": true, + "tool_choice": "auto", + "tools": [] + }, + "stateful_encrypted": { + "id": "resp_stateful_encrypted_001", + "object": "response", + "created_at": 1738531300, + "model": "gpt-5", + "status": "completed", + "previous_response_id": "resp_litellm_encoded_prev_id", + "output": [ + { + "type": "reasoning", + "id": "rsn_100", + "status": "completed", + "summary": [ + { "type": "summary_text", "text": "Computed product and formed concise explanation." } + ], + "encrypted_content": "A1B2C3D4E5F6...encrypted bytes..." + }, + { + "type": "message", + "id": "msg_100", + "status": "completed", + "role": "assistant", + "content": [ + { "type": "output_text", "text": "1234 × 5678 = 7,006,652. It's the product of the two inputs." } + ] + } + ], + "parallel_tool_calls": true, + "tool_choice": "auto", + "tools": [] + } +}