Skip to content

Add SQLModel ORM layer alongside raw SQLite for database portability#9071

Draft
Pfannkuchensack wants to merge 7 commits intomainfrom
feature/sqlmodel-migration
Draft

Add SQLModel ORM layer alongside raw SQLite for database portability#9071
Pfannkuchensack wants to merge 7 commits intomainfrom
feature/sqlmodel-migration

Conversation

@Pfannkuchensack
Copy link
Copy Markdown
Collaborator

Summary

Introduces SQLModel (SQLAlchemy + Pydantic) as an ORM layer to enable future database backend switching (PostgreSQL, MySQL). All services except session_queue have been migrated to SQLModel-based implementations while keeping the existing migration system and raw SQLite connection intact for backwards compatibility.

Key changes:

  • Add sqlmodel dependency
  • Define SQLModel table models for all 14 database tables
  • Extend SqliteDatabase with SQLAlchemy Engine and Session management
  • Create SQLModel implementations for 10 services (boards, images, workflows, models, users, style presets, app settings, etc.)
  • Session queue remains on raw SQLite (Phase 3)
  • Add 95 unit tests and 12 performance benchmarks
  • Optimize with StaticPool, expire_on_commit=False, and read-only sessions

Related Issues / Discussions

This PR is the first phase of a larger migration proposal to make InvokeAI's database layer portable across different backends (SQLite, PostgreSQL, MySQL).

QA Instructions

Run the new unit tests

uv run --extra cuda --extra test python -m pytest tests/app/services/test_sqlmodel_services/ -v --timeout=30

Expected: 94 passed, 1 skipped

Run the performance benchmarks

uv run --extra cuda --extra test python -m pytest tests/app/services/test_sqlmodel_services/test_benchmark_sqlmodel_vs_sqlite.py -v -s --timeout=120

Expected: 12 passed with timing comparison output

Manual testing

  1. Start InvokeAI normally — all existing functionality should work as before
  2. Create/edit/delete boards and verify they persist
  3. Generate images and verify they appear in the gallery
  4. Create/manage style presets
  5. Verify user authentication works (if multiuser is enabled)

What changed at runtime

  • dependencies.py now instantiates SQLModel-based services instead of raw SQLite ones
  • The SqliteDatabase class now also provides a SQLAlchemy Engine + Session alongside the existing raw sqlite3 connection
  • The raw connection is still used for migrations and the session queue
  • All other services use SQLModel for queries

Merge Plan

  • This PR does not change the database schema — migrations remain untouched
  • The old raw SQLite implementations are kept as-is (not deleted) for reference and rollback
  • Session queue migration is planned as Phase 3 in a future PR
  • No special merge timing required

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

Introduces SQLModel (SQLAlchemy + Pydantic) as an ORM layer to enable
future database backend switching (PostgreSQL, MySQL). All services
except session_queue have been migrated to SQLModel-based implementations
while keeping the existing migration system and raw SQLite connection
intact for backwards compatibility.

Key changes:
- Add sqlmodel dependency
- Define SQLModel table models for all 14 database tables
- Extend SqliteDatabase with SQLAlchemy Engine and Session management
- Create SQLModel implementations for 10 services (boards, images,
  workflows, models, users, style presets, app settings, etc.)
- Session queue remains on raw SQLite (Phase 3)
- Add 95 unit tests and 12 performance benchmarks
- Optimize with StaticPool, expire_on_commit=False, and read-only sessions
@github-actions github-actions Bot added api python PRs that change python files Root services PRs that change app services python-tests PRs that change python tests python-deps PRs that change python dependencies labels Apr 19, 2026
Port SqliteSessionQueue to a SQLAlchemy Core / SQLModel hybrid that keeps the
existing public API and DB schema (migrations and triggers untouched). Hot
paths (enqueue bulk insert, dequeue, bulk cancel/delete, list with cursor
pagination, status aggregations) use Core to avoid ORM hydration overhead;
single-row reads stay ORM-style for clarity.

- Add SqlModelSessionQueue alongside the legacy SqliteSessionQueue
- Add the missing `workflow` column to SessionQueueTable (was added by
  migration_2 but never declared on the SQLModel)
- Wire dependencies.py to the new implementation
- Add 36 unit tests covering enqueue/dequeue, status mutations, bulk
  cancel/delete, prune-to-limit, retry, pagination and aggregations
- Avoid nested write sessions on the single StaticPool connection by reading
  the current item before opening the outer write session
@lstein lstein added the 6.14.x label Apr 25, 2026
@lstein lstein moved this to 6.14.x Theme: LIBRARY UPDATES in Invoke - Community Roadmap Apr 25, 2026
@JPPhoto
Copy link
Copy Markdown
Collaborator

JPPhoto commented Apr 26, 2026

Why is database portability a significant issue?

@Pfannkuchensack
Copy link
Copy Markdown
Collaborator Author

Pfannkuchensack commented Apr 26, 2026

If you want to host the multi User Feature with many User there is a Problem with Performance on sqlite. (this is a PR to test it)

@JPPhoto
Copy link
Copy Markdown
Collaborator

JPPhoto commented Apr 26, 2026

If you want to host the multi User Feature with many User there is a Problem with Performance on sqlite. (this is a PR to test it)

It would be interesting to capture some real-world Invoke sqlite3 access patterns and benchmark those. Perhaps that would tell us if there's first an opportunity for optimization in the way we do things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.14.x api python PRs that change python files python-deps PRs that change python dependencies python-tests PRs that change python tests Root services PRs that change app services

Projects

Status: 6.14.x Theme: LIBRARY UPDATES

Development

Successfully merging this pull request may close these issues.

3 participants