feat(ai): validate token permissions before running evaluations

[islameldigwi]

Overview

Checks that the token has both write (ingest traces) and read (query dataset) permissions before starting evals, preventing wasted time on permission-related failures. Provides clear, actionable error messages with token management links.

Showcase

• Missing write permissions:

• Missing read permissions:

---

Note

Medium Risk
Adds a new preflight network call that can fail/exit before running evals and changes connection resolution by deriving an edgeRegion from edgeUrl, which could impact eval execution in different environments.

Overview
Preflights axiom eval runs by calling a new /api/v3/evaluations/validate endpoint to ensure the configured token can write (ingest) and read (query) for the target dataset; on failure it prints actionable, formatted guidance (including console links) and exits early (skipped in --debug).

Extends config resolution to accept eval.edgeUrl (example updated) and derives an edgeRegion from the edge hostname (defaulting to us-east-1, with localhost/non-edge fallbacks). Improves eval lifecycle robustness by capturing instrumentation init failures separately for reporting and hardening registration/update error handling, plus adds Vitest coverage for token validation and edge region resolution.

^{Written by Cursor Bugbot for commit e126238. This will update automatically on new commits. Configure here.}

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/axiom@244

commit: 4870c1d

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[cursor]

instrumentationError overwritten before reporter reads it

High Severity

The instrumentationError property is set on suite.meta.evaluation at line 261, but a few lines later the entire suite.meta.evaluation object is reassigned at line 344 without including instrumentationError. This means the value is always lost, and the reporter (which reads meta.evaluation.instrumentationError to display warnings) will never see it. The new instrumentation error reporting feature in the reporter and console-utils is effectively dead code.

Additional Locations (1)

• packages/ai/src/evals/eval.ts#L260-L261

 

[cursor]

Debug console.log statements left in production code

Medium Severity

console.log({ region: connection.edgeRegion }) on line 78 prints a debug object to stdout on every eval run. console.debug('validation response', { data }) on line 91 logs error response payloads. Both appear to be leftover debugging statements that will produce unexpected noisy output for users running evaluations.

Additional Locations (1)

• packages/ai/src/config/validate-eval-token-permissions.ts#L90-L91