feat: apply select enrichment attributes from a single source (CM-1355)

[skwowet]

Summary

Single-source enrichment can now apply a small allowlisted attribute set (location, country today) so aggregate downstream consumers get better coverage without relaxing the multi-source quality bar for a full profile update.

processMemberSources was also refactored so array-shaped sources (e.g. LinkedIn scrapers) are resolved the same way regardless of apply mode — pick one profile first, then apply.

Structure

Mental model after this change:

normalize caches
  → resolve arrays to a single profile (shared)
  → apply by post-resolve source count
       1 source  → allowlisted attributes only
       2+ sources & activity > 100 → full squash
       otherwise → touch lastTriedAt

1. Normalize — load enrichment caches into toBeSquashed (object or array per source).
2. Resolve — any array-shaped source goes through the existing profile-selection activity (LLM when needed). Selected profile replaces the array; no selection drops the source; discarded LinkedIn identities still cascade-remove matching sources. Order preserved: Crustdata → ProgAI LinkedIn scraper → other arrays.
3. Apply — decided from what remains after resolve:
   • 1 source → write only SINGLE_SOURCE_ENRICHMENT_ATTRIBUTES
   • 2+ sources & activity > 100 → existing full squash (identities, attributes, orgs, reach)
4. Fallback — touch lastTriedAt when neither apply path runs.

Attribute reads use any enrichment-* key on the attribute (not hard-coded to enrichment-${source}), so wrapped normalizers (e.g. scraper → ProgAI keys) still work.

Changes

• Add SINGLE_SOURCE_ENRICHMENT_ATTRIBUTES and pass activityCount into processMemberSources
• Always call processMemberSources on enrichment cache changes; move the activity gate into the multi-source full-squash path
• Shared array→profile resolve before apply (removes duplicated Crustdata / ProgAI scraper blocks)
• Single-source path applies allowlisted attributes after resolve
• Multi-source squash uses the same enrichment-attribute lookup helper

[Copilot]

Pull request overview

Enables limited member enrichment from a single source while retaining the multi-source quality threshold.

Changes:

• Allow location and country updates from one source.
• Move the activity-count gate to full multi-source squashing.
• Pass activity counts into source processing.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
processMemberSources.ts	Adds single-source handling and multi-source gating.
enrichMember.ts	Processes all cache changes and passes activity count.
config.ts	Defines safe single-source attributes.
types.ts	Adds activity count to workflow arguments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

services/apps/members_enrichment_worker/src/workflows/processMemberSources.ts:42

• The helper returns unknown, but its result is passed to cleanAttributeValue, whose parameter is a string, string array, or record. TypeScript rejects that call because unknown is not assignable to the activity parameter type. Preserve the activity's accepted type here (or narrow the value before calling it).

): unknown {

services/apps/members_enrichment_worker/src/workflows/processMemberSources.ts:107

• This gate checks the source count before array resolution. For a member with activityCount <= 100 and two cached sources, if an array source resolves to no profile, the remaining single source should enter the allowlisted path, but resolution is skipped and the attributes are never applied. Resolve arrays first, then choose the apply path from sourceKeys.

  // Only resolve arrays when an apply path can run afterward.
  if (
    arraySources.length > 0 &&
    (Object.keys(toBeSquashed).length === 1 || args.activityCount > 100)

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

services/apps/members_enrichment_worker/src/workflows/processMemberSources.ts:107

• This gate uses the pre-resolve source count, so a low-activity member with two cached sources never resolves its array source. If profile selection drops that array, the remaining object source should enter the single-source allowlist path, but the workflow instead falls through to lastTriedAt. Resolve arrays before deciding the apply mode, as described by the post-resolve source-count model.

  // Only resolve arrays when an apply path can run afterward.
  if (
    arraySources.length > 0 &&
    (Object.keys(toBeSquashed).length === 1 || args.activityCount > 100)
  ) {

services/apps/members_enrichment_worker/src/workflows/processMemberSources.ts:154

• Array sources later in orderedArraySources are still unresolved here, so skipping them prevents a discarded LinkedIn identity from cascading to a matching scraper profile. That rejected profile can subsequently be selected and retained as another source. Filter/delete matching profiles in unresolved arrays, or defer the cascade until every array has been resolved.

          for (const otherSource of Object.keys(toBeSquashed)) {
            const profile = toBeSquashed[otherSource]
            if (Array.isArray(profile)) {
              continue

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

services/apps/members_enrichment_worker/src/workflows/processMemberSources.ts:155

• Skipping unresolved array sources breaks the discarded-identity cascade across the ordered sources. A profile discarded while resolving Crustdata can still be selected later from the ProgAI scraper array (a lone verified profile is selected without LLM matching), reintroducing an identity already rejected earlier. Remove matching profiles from unresolved arrays as well, and delete the source if none remain.

            const profile = toBeSquashed[otherSource]
            if (Array.isArray(profile)) {
              continue
            }

services/apps/members_enrichment_worker/src/workflows/processMemberSources.ts:107

• This gate uses the pre-resolution source count, so it can skip the very resolution needed to discover the single-source apply path. For example, with two cached sources and activityCount <= 100, an array source that would resolve to no profile is never dropped; the remaining object's allowlisted attributes are therefore not applied even though apply mode is defined by the post-resolve count. Resolve arrays before evaluating either apply path.

This issue also appears on line 152 of the same file.

  // Only resolve arrays when an apply path can run afterward.
  if (
    arraySources.length > 0 &&
    (Object.keys(toBeSquashed).length === 1 || args.activityCount > 100)
  ) {