Skip to content

Deduplicate indexing filters via a data-provider membership check#179

Open
loevgaard wants to merge 1 commit into
masterfrom
issue-178-deduplicate-indexing-filters
Open

Deduplicate indexing filters via a data-provider membership check#179
loevgaard wants to merge 1 commit into
masterfrom
issue-178-deduplicate-indexing-filters

Conversation

@loevgaard

Copy link
Copy Markdown
Member

Closes #178.

Problem

"What should be indexed" lived in two parallel mechanisms that had to be kept in sync by hand:

  1. Query-builder (SQL) filters (EventSubscriber/IndexableDataFilter/*) — shape a full reindex, applied inside DefaultIndexableDataProvider::getIds().
  2. Object filters (Filter/Entity/*) — applied per entity in DefaultIndexer::indexEntities(), on both the batch and the incremental path.

EnabledEntityFilter/StockAvailableEntityFilter/ChannelsAwareEntityFilter mirrored the SQL filters so the same rules also applied on the incremental (Doctrine-event) path, where the query builder never runs. If a SQL/PHP pair diverged, batch and incremental disagreed about what qualifies — and since the incremental path issues deleteDocuments on filter-out, a divergence could delete a document a reindex would keep.

Solution

Route the incremental path through the same data provider the batch path uses, making the query-builder filters the single source of truth.

  • IndexableDataProviderInterface gains containsId(string $entity, Index $index, int|string $id): bool.
  • DefaultIndexableDataProvider builds its query once in a private createQueryBuilder() shared by getIds() and containsId(). The id constraint is added after the QueryBuilderForDataProvisionCreated dispatch, so every filter subscriber sees the identical query on both paths — that's what keeps them consistent.
  • IndexEntityHandler asks containsId() whether a changed entity still qualifies for each index: index it if so, removeEntity() (delete its document) if not.

Only the two true mirrors are removed (EnabledEntityFilter, StockAvailableEntityFilter) along with their object-filter registrations in the extension. ChannelsAwareEntityFilter (per-scope channel membership — the SQL twin only checks SIZE(channels) > 0) and FilterableEntityFilter (arbitrary user FilterableInterface) stay. EntityFilterInterface remains the documented escape hatch for rules that can't be expressed in SQL or that depend on the channel/locale/currency scope.

BC breaks (pre-1.0)

  • New method on IndexableDataProviderInterface — custom data providers must implement containsId().
  • Filter\Entity\EnabledEntityFilter and Filter\Entity\StockAvailableEntityFilter are removed.

Notes

  • Trade-off: the incremental path now runs one small indexed WHERE id = ? query per change per index instead of a purely in-memory check.
  • Eventual consistency is unchanged: indexing runs through Messenger, so a document can briefly lag its entity — documented in the README.

Tests / QA

  • New IndexEntityHandlerTest (index-when-contained, remove-when-not, not-found, non-scalar id).
  • New DefaultIndexableDataProviderTest (functional): proves getIds()/containsId() agree and that disabling a product flips containsId() to false.
  • Deleted the two mirror-filter unit tests with their classes.
  • Green: ECS, PHPStan (level max, no baseline), Rector dry-run, composer validate --strict, composer normalize --dry-run, lint:container, Unit (111) and Functional (17) suites.
  • End-to-end smoke (throwaway test, since removed): disabling a fixture product through the ORM fired the Doctrine listener, removed its document from Meilisearch, and re-enabling re-indexed it.

"What should be indexed" was expressed twice: query-builder (SQL) filters
that shape a full reindex, and mirror object filters applied per entity on
both the batch and incremental paths. Keeping the two in sync was manual,
and a divergence could delete a document a reindex would keep.

Route the incremental (single-entity) path through the same data provider
the batch path uses. IndexableDataProviderInterface gains containsId(),
implemented via a query builder shared with getIds() (same
QueryBuilderForDataProvisionCreated event, constrained to one id), so the
SQL filters are the single source of truth. IndexEntityHandler asks
containsId() whether a changed entity still qualifies: index it if so,
remove its document if not.

The two true mirrors (EnabledEntityFilter, StockAvailableEntityFilter) are
removed along with their object-filter registrations. ChannelsAwareEntityFilter
and FilterableEntityFilter stay, since they carry per-scope / SQL-inexpressible
logic; EntityFilterInterface remains the escape hatch for such rules.

BC breaks (pre-1.0): new interface method on IndexableDataProviderInterface,
and removal of the two filter classes.

Closes #178
@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 42.85714% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.03%. Comparing base (a615526) to head (c363346).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
src/DataProvider/DefaultIndexableDataProvider.php 0.00% 12 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master     #179      +/-   ##
============================================
- Coverage     49.08%   49.03%   -0.05%     
+ Complexity      805      798       -7     
============================================
  Files           139      137       -2     
  Lines          2557     2541      -16     
============================================
- Hits           1255     1246       -9     
+ Misses         1302     1295       -7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deduplicate indexing filters: run the single-entity path through the batch data provider

1 participant