NOTE: Don't assign yourself unless you have have confirmed with Matthew you've got a working environment
🧠 Context
The last_synced_at field on a source is intended to track when it was most recently ingested. However, this field is currently not updated after ingestion runs, which makes it harder to determine whether chunks are stale or current.
This field needs to be updated immediately after a source is successfully ingested to ensure downstream logic (like stale chunk cleanup) works correctly.
🛠 Implementation Plan
-
In src/ingestion/services/webpage_ingestion_service.py, after successfully processing and storing chunks:
- Set
source.last_synced_at = datetime.utcnow().
- Save the updated
source to the database.
-
Ensure this happens before triggering the cleanup task.
-
Add a test to confirm that last_synced_at is updated after ingestion.
✅ Acceptance Criteria
- After ingestion completes successfully for a source, update
source.last_synced_at to the current timestamp.
- Save the updated source object to persist the change.
- This should happen in
WebpageIngestionService after all chunks are created and saved.
NOTE: Don't assign yourself unless you have have confirmed with Matthew you've got a working environment
🧠 Context
The
last_synced_atfield on a source is intended to track when it was most recently ingested. However, this field is currently not updated after ingestion runs, which makes it harder to determine whether chunks are stale or current.This field needs to be updated immediately after a source is successfully ingested to ensure downstream logic (like stale chunk cleanup) works correctly.
🛠 Implementation Plan
In
src/ingestion/services/webpage_ingestion_service.py, after successfully processing and storing chunks:source.last_synced_at = datetime.utcnow().sourceto the database.Ensure this happens before triggering the cleanup task.
Add a test to confirm that
last_synced_atis updated after ingestion.✅ Acceptance Criteria
source.last_synced_atto the current timestamp.WebpageIngestionServiceafter all chunks are created and saved.