Skip to content

feat: stamp automation runs with source provenance - #437

Open
palrohitg wants to merge 7 commits into
OpenHands:mainfrom
palrohitg:feat/342-run-source-provenance
Open

palrohitg wants to merge 7 commits into
OpenHands:mainfrom
palrohitg:feat/342-run-source-provenance

Conversation

@palrohitg

@palrohitg palrohitg commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Why

When an automation changes after a successful run, the run history currently cannot answer a basic operational question: which source did that run execute? automation_runs stores the outcome, timestamps, cost, and execution identifiers, but source remains only on the mutable automations row. Reading automation.tarball_path later therefore describes the automation now, not the historical run.

That missing provenance blocks the first, independent step of rollback work proposed in #342: identifying the last known-good source before deciding how to restore or pin it.

📄 Open the visual design note

Summary

This PR makes source provenance a property of each AutomationRun:

  • Adds nullable source_tarball_path and source_commit columns through cross-database migration 023.
  • Initializes both fields when scheduler- and webhook-triggered runs are created.
  • Refreshes and freezes the snapshot in the same transaction that claims a PENDING run as RUNNING. This preserves the service's existing behavior—an edit made while a run is queued applies when it is dispatched—while ensuring the recorded source matches what execution receives.
  • Changes _execute_run to use the run's frozen source_tarball_path, with a fallback to automation.tarball_path for legacy rows.
  • Copies AutomationGitSyncState.last_synced_commit only when the sync state is clean. A dirty state intentionally records source_commit = null, because the previous SHA no longer identifies the current DB-side tarball.
  • Exposes both fields on AutomationRunResponse without changing existing response fields.

Behaviour

Automation source state when dispatch claims the run source_tarball_path source_commit
Git-managed and fully reconciled Frozen path selected for execution Last reconciled commit SHA
Git-managed but dirty after a DB/API edit Frozen path selected for execution null—the old SHA would be misleading
Not managed by Git / never synced Frozen path selected for execution null
Historical row created before migration null null

The nullable commit is deliberate: an absent provenance claim is safer than a precise-looking but stale SHA.

Dispatch consistency

The source is initialized at run creation for immediate visibility, but creation alone is not sufficient because a run may remain PENDING while the automation is edited.

The claim flow is now:

  1. Poll a PENDING run together with its automation.
  2. Resolve the automation's current tarball and trustworthy clean Git commit.
  3. Write those values onto the run.
  4. Transition the same run to RUNNING in the same database transaction.
  5. Execute run.source_tarball_path, not the mutable automation field.

This closes the edit-while-pending race without changing which version the dispatcher selects. Once claimed, a later automation edit cannot change the tarball used by that run.

API and migration

AutomationRunResponse gains two additive optional fields:

{
  "source_tarball_path": "oh-internal://uploads/2fbd...",
  "source_commit": "4a42e3c9..."
}

Compatibility properties:

  • The migration uses generic SQLAlchemy Text and String(64) types and follows the repository's PostgreSQL/SQLite pattern.
  • Both columns are nullable, so existing rows require no backfill and retain their honest “unknown” provenance.
  • PostgreSQL receives descriptive column comments; SQLite skips only those comments.
  • Existing API clients remain compatible because the response change is additive.
  • Runs created before this deployment remain executable through the legacy tarball fallback.

Changes by area

Area Change
models.py / migration 023 Persist run-level tarball and Git provenance.
utils/run.py Centralize snapshot logic and reject dirty/stale Git commits.
utils/webhook.py Apply the same provenance semantics to event-triggered runs.
dispatcher.py Freeze provenance at claim time and execute the frozen tarball.
schemas.py Return provenance through the run API.
Tests Cover clean, dirty, non-Git, serialization, migration, pending-edit, and frozen-execution cases.

Scope

This is intentionally phase 1 of #342, which the issue identifies as independently useful.

Included:

  • Answering “which tarball and trustworthy Git commit was selected for this run?”
  • Preserving that answer after later automation edits.

Not included:

  • Parsing or storing a Git URL with a branch/tag/commit ref.
  • Pinning Git sync to a historical ref.
  • A restore-last-good endpoint or automatic rollback policy.
  • Agent Canvas UI for displaying or restoring a version.
  • A complete immutable execution manifest: entrypoint, model, secrets, and runtime dependencies remain outside this source-provenance snapshot.

The PR therefore references, but does not close, #342.

How to test

Automated checks completed locally:

  • uv run pre-commit run --files <all changed Python and migration files>
    • Ruff format: passed
    • Ruff lint: passed
    • pycodestyle: passed
    • Pyright: passed
  • uv run pytest tests/test_telemetry.py tests/test_schemas.py tests/test_db.py::TestSqliteMigrations::test_migrations_run_on_sqlite -q
    • 41 passed, with one existing get_settings() deprecation warning
  • SQLite end-to-end dispatch exercise:
    1. Create a PENDING run carrying an old source.
    2. Claim it against a clean, newer automation source and commit.
    3. Edit the automation again after the claim.
    4. Execute the run through _execute_run with the execution backend mocked at the external boundary.
    5. Assert execution receives the source frozen at claim time.
    • Passed
  • git diff --check: passed
  • GitHub PR description and conventional-title checks: passed

Not run locally:

  • PostgreSQL Testcontainers tests: Docker Desktop was installed but its daemon did not become available in this environment.
  • Live staging sandbox E2E: no OPENHANDS_API_KEY was available. This does not affect the local source-selection path above; it prevents authentication to the external staging sandbox service.

Issue Number

Implements the run-version stamping phase of #342. Follow-up work is required before the overall rollback proposal is complete.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

@all-hands-bot

Copy link
Copy Markdown
Contributor

👋 This PR needs a couple of things fixed before OpenHands can review it:

  • the PR description's HUMAN: section needs at least 20 characters describing what you tested, not just the template placeholder

Push an update once this is addressed and this check re-runs automatically.

This is an automated check - no AI was used to generate this comment.

@github-actions github-actions Bot added the type: feat A new feature label Sep 10, 2026
@palrohitg

palrohitg commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

@VascoSch92 @DevinVinson @enyst could one of you approve the fork workflows and take a look when you have a chance?

This PR covers phase 1 of #342 (run source stamping). The test and CI workflows are waiting for first-time contributor approval and haven’t started yet. Thanks!

@github-actions

Copy link
Copy Markdown
Contributor

📁 PR Artifacts Notice

This PR contains a .pr/ directory with temporary PR-specific documents. Because this is a fork PR, the workflow will open or update a cleanup PR against main after merge.

@palrohitg

Copy link
Copy Markdown
Contributor Author

@tofarr If can you help here ? to trigger the UT.

@palrohitg

Copy link
Copy Markdown
Contributor Author

The failed unit-test job is a shared CI dependency failure, not a provenance assertion failure: 1,703 tests passed, then all 17 S3 integration cases failed in fixture setup because Docker Hub no longer serves the pinned MinIO image. The registry repair is isolated in #447.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: feat A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants