Add SQLModel ORM layer alongside raw SQLite for database portability#9071
Draft
Pfannkuchensack wants to merge 7 commits intomainfrom
Draft
Add SQLModel ORM layer alongside raw SQLite for database portability#9071Pfannkuchensack wants to merge 7 commits intomainfrom
Pfannkuchensack wants to merge 7 commits intomainfrom
Conversation
Introduces SQLModel (SQLAlchemy + Pydantic) as an ORM layer to enable future database backend switching (PostgreSQL, MySQL). All services except session_queue have been migrated to SQLModel-based implementations while keeping the existing migration system and raw SQLite connection intact for backwards compatibility. Key changes: - Add sqlmodel dependency - Define SQLModel table models for all 14 database tables - Extend SqliteDatabase with SQLAlchemy Engine and Session management - Create SQLModel implementations for 10 services (boards, images, workflows, models, users, style presets, app settings, etc.) - Session queue remains on raw SQLite (Phase 3) - Add 95 unit tests and 12 performance benchmarks - Optimize with StaticPool, expire_on_commit=False, and read-only sessions
…ture/sqlmodel-migration
Port SqliteSessionQueue to a SQLAlchemy Core / SQLModel hybrid that keeps the existing public API and DB schema (migrations and triggers untouched). Hot paths (enqueue bulk insert, dequeue, bulk cancel/delete, list with cursor pagination, status aggregations) use Core to avoid ORM hydration overhead; single-row reads stay ORM-style for clarity. - Add SqlModelSessionQueue alongside the legacy SqliteSessionQueue - Add the missing `workflow` column to SessionQueueTable (was added by migration_2 but never declared on the SQLModel) - Wire dependencies.py to the new implementation - Add 36 unit tests covering enqueue/dequeue, status mutations, bulk cancel/delete, prune-to-limit, retry, pagination and aggregations - Avoid nested write sessions on the single StaticPool connection by reading the current item before opening the outer write session
…ture/sqlmodel-migration
…e-ai/InvokeAI into feature/sqlmodel-migration
Collaborator
|
Why is database portability a significant issue? |
Collaborator
Author
|
If you want to host the multi User Feature with many User there is a Problem with Performance on sqlite. (this is a PR to test it) |
Collaborator
It would be interesting to capture some real-world Invoke sqlite3 access patterns and benchmark those. Perhaps that would tell us if there's first an opportunity for optimization in the way we do things. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces SQLModel (SQLAlchemy + Pydantic) as an ORM layer to enable future database backend switching (PostgreSQL, MySQL). All services except session_queue have been migrated to SQLModel-based implementations while keeping the existing migration system and raw SQLite connection intact for backwards compatibility.
Key changes:
Related Issues / Discussions
This PR is the first phase of a larger migration proposal to make InvokeAI's database layer portable across different backends (SQLite, PostgreSQL, MySQL).
QA Instructions
Run the new unit tests
uv run --extra cuda --extra test python -m pytest tests/app/services/test_sqlmodel_services/ -v --timeout=30Expected: 94 passed, 1 skipped
Run the performance benchmarks
uv run --extra cuda --extra test python -m pytest tests/app/services/test_sqlmodel_services/test_benchmark_sqlmodel_vs_sqlite.py -v -s --timeout=120Expected: 12 passed with timing comparison output
Manual testing
What changed at runtime
dependencies.pynow instantiates SQLModel-based services instead of raw SQLite onesSqliteDatabaseclass now also provides a SQLAlchemy Engine + Session alongside the existing rawsqlite3connectionMerge Plan
Checklist
What's Newcopy (if doing a release after this PR)