Skip to content

refactor(agent): clarify model, provider, protocol, auth, and applied config semantics #196

Description

@zpzjzj

Background

Agent configuration currently mixes several independent concepts:

  • the agent CLI (engine);
  • the adapter wire protocol;
  • the opaque upstream model identifier;
  • the provider namespace used for credentials and endpoints;
  • injected authentication versus delegated agent-local login;
  • values requested by the user, values applied by skill-up, and values actually observed from the running agent.

The problem is not the name or YAML nesting of provider. The problem is that provider, protocol, routing, credentials, and runtime observation are resolved in multiple layers and can be silently rewritten or reported as if they were equivalent.

Local login remains a supported workflow. An already logged-in Codex CLI, QoderCLI, Claude Code, or Qwen Code must be usable without injected auth. Preflight checks must remain static and must not make a model request or consume tokens.

Intended semantics

engine                 selects the CLI adapter and its default protocol
model.name             is an opaque upstream model identifier
model.provider         selects a credential/endpoint namespace
auth mode              is injected or delegated to agent-local state
requested config       is selected from YAML/CLI/env/credential inputs
applied config         is what skill-up forwards after adapter normalization
observed config        is only what the running agent explicitly reports

The engine, not the provider, determines the protocol. For example, provider: dashscope with Codex selects an OpenAI-compatible connection, while the same provider with Claude Code selects an Anthropic-compatible connection.

Missing observed values are unknown. Applied configuration must never be presented as proof of the provider, model, credential, or local login ultimately used by the agent.

Current problems

  1. provider is overloaded as a legacy model prefix, credential namespace, apparent vendor, and adapter input.
  2. --model X/Y is ambiguous between a legacy provider/model form and an opaque slashed model ID.
  3. Explicit configuration can be silently overridden or ignored (<PROVIDER>_MODEL, unsupported Qoder modes, Codex custom-provider fallback, ineffective kwargs).
  4. Credential files can describe protocol-specific endpoints, but current resolution flattens them before the adapter protocol is known.
  5. Schema fields such as engine.version, engine.entry, and engine.model.params are not consistently consumed.
  6. Installation checks, static auth resolution, and real authentication validation do not have a clear lifecycle boundary.
  7. Runner and agent_judge must share the same primitives without inferring an engine from a provider or duplicating compatibility branches.

Model connection boundary

Harbor's recent model-connection work is a useful layering reference:

We should borrow the separation, not Harbor's provider inference or credential alias fan-out:

  1. Provider configuration is static input and may contain protocol-specific endpoints.
  2. Resolved model connection selects one (provider, protocol) connection and retains value sources plus auth/routing mode.
  3. Adapter materialization turns the connection into flags, env, or agent-native configuration.
  4. Observed configuration remains separate and is populated only from an explicit agent report.

A future internal shape may be:

type ResolvedModelConnection struct {
    Provider      string
    Protocol      Protocol
    APIKey        string
    BaseURL       string
    AuthMode      AuthMode    // injected or agent-local
    RoutingMode   RoutingMode // explicit, native config, agent-local, unresolved
    APIKeySource  ValueSource
    BaseURLSource ValueSource
}

Secrets must not be serialized into reports. We should preserve the selected source instead of exporting one credential under unrelated aliases. Where configuration supports inheritance, absent and explicit-empty values may also need distinct representation.

Agent-native config files and local login databases are owned by the agent and may be opaque to skill-up. Native config support, if introduced, must sit after connection resolution rather than becoming a second provider/model/auth resolver.

Explicit provider and legacy compatibility

Add an explicit CLI option:

skill-up run ... --provider dashscope --model qwen3.6-plus

Rules:

  1. Explicit --provider overrides YAML engine.model.provider.
  2. With explicit --provider, the complete --model value is opaque and is never split on /.
  3. Without --provider, retain the historical --model provider/name compatibility path.
  4. Split a legacy prefix only when it is a known/configured provider; otherwise preserve the complete slashed model ID.
  5. Existing v1alpha1 YAML with separate provider and model fields remains supported.
  6. Released GitHub Actions and historical CLI examples continue to work without modification.

Judge Agent consideration

This issue does not add an independent Judge Agent schema. It establishes common runner/judge primitives so a later explicit judge engine can reuse the same merge, connection resolution, capability validation, adapter materialization, runtime provisioning, and reporting flow.

Until that schema is agreed, the judge may continue inheriting the runner engine for compatibility. A provider namespace must never be used to infer a judge engine. Judge-specific kwargs must not accidentally inherit incompatible runner-only settings.

Non-goals

  • Do not require API keys when an agent supports local login.
  • Do not validate auth with a token-consuming preflight request.
  • Do not modify a user's host-installed agent under runtime: none.
  • Do not break v1alpha1 YAML or move provider merely for naming purity.
  • Do not infer an engine or provider from an opaque model ID when an explicit value is available.
  • Do not add native agent configuration support to the current capability PR.

Phased PR plan

PR 1: Characterization and terminology

  • Capture adapter behavior, historical slash forms, GitHub Action translation, local-login delegation, and opaque model IDs.
  • Define provider, protocol, requested, applied, and observed semantics.

PR 2: Central role-aware agent configuration — completed in #211

  • Build one role-aware ResolvedAgentConfig after YAML/CLI merge and credential loading.
  • Remove repeated parse/mutate/collapse/normalize behavior from callers.
  • Preserve compatibility while retaining value sources.

PR 3: Adapter capabilities and requested/applied/observed reporting — completed in #217

  • Declare protocol and supported model/endpoint/kwarg behavior per adapter.
  • Normalize once before execution and warn on unsupported explicit settings.
  • Report credential-free requested/applied values and only explicit observations.
  • Keep local/default selection unknown and do not add auth probes.

PR 4a: Protocol-aware model connection primitives — completed in #221

  • Introduce provider configuration and resolved connection types.
  • Resolve endpoints and credentials by (provider, protocol).
  • Retain sources and auth/routing modes, including delegated local state.
  • Define absent versus explicit-empty semantics where inheritance requires it.
  • Add resolver tests without changing public behavior.

PR 4b: Adapter materialization migration — completed in #226

  • Make factory/adapters consume the resolved model connection.
  • Materialize flags, env, and native configuration through one adapter boundary.
  • Do not fan credentials out to unrelated aliases or inject credentials for a provider the adapter rejected.
  • Preserve local-login delegation.

PR 4c: Explicit provider and compatibility migration — completed in #222

  • Add --provider and treat the model as opaque when it is explicit.
  • Retain legacy --model provider/name behavior.
  • Change provider-scoped model values to defaults that do not override explicit YAML/CLI models.
  • Cover the root GitHub Action and historical CLI/YAML forms.

PR 5: Installation and version lifecycle — in #233

  • Run side-effect-free binary/version checks before case execution.
  • Define consistent host versus isolated-runtime version semantics.
  • Record the detected version as observed metadata.
  • Keep auth validation out of installation checks.

PR 6: Schema/native-config decisions and Judge follow-up

Acceptance criteria

  • Existing v1alpha1 YAML, released GitHub Actions, and --model provider/name calls remain supported.
  • --provider dashscope --model a/b deterministically preserves a/b.
  • Unknown slashed model IDs remain opaque when no provider can be resolved.
  • Explicit models are not silently overridden by provider-scoped defaults.
  • Connections select credentials/endpoints by (provider, protocol) without cross-provider alias fan-out.
  • A rejected provider does not leak its endpoint or credential into agent-local fallback.
  • Codex CLI and QoderCLI continue to work with local login and no injected key.
  • No installation/auth preflight consumes model tokens.
  • Unsupported explicit configuration is surfaced before case execution.
  • Reports distinguish requested, applied, and observed configuration; unknown runtime state remains unknown.
  • Runner and judge can share the same primitives without provider-to-engine inference or duplicated compatibility logic.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions