Fix D15: match Clojure moderation handling (zero out columns, don't remove)

Python's _apply_moderation() removed moderated-out comment columns entirely
from rating_mat. Clojure's zero-out-columns (named_matrix.clj:214-230) sets
all values in moderated columns to 0, preserving matrix structure.

Change _apply_moderation() to zero out moderated columns instead of removing
them, so that:
- rating_mat retains the same column count as raw_rating_mat
- tids output includes moderated-out tids (matching Clojure)
- Matrix dimensions are preserved through the pipeline

Moderated-out participants (rows) are still removed — unchanged.

Zeroed columns have no signal (na=0, nd=0), so they fail all significance
tests and are effectively excluded from repness/consensus/PCA, but their
presence preserves index stability for downstream consumers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: 2 people

delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md

@@ -711,6 +711,60 @@ to create a new worktree. If yes, provide a prompt they can use to start that se
 
 ---
 
+## Session: Fix D15 — Moderation Handling (2026-03-16)
+
+### Branch: `jc/clj-parity-d15-moderation-handling-zeros-vs-removes`
+
+### What was done
+
+Fixed D15: Python now zeros out moderated-out comment columns instead of removing them,
+matching Clojure's `zero-out-columns` behavior (named_matrix.clj:214-230).
+
+**The discrepancy**: Python's `_apply_moderation()` removed moderated-out columns from
+`rating_mat` entirely (`raw_rating_mat.loc[keep_ptpts, keep_comments]`). Clojure zeros
+them out (`matrix/set-column m' i 0`), preserving matrix structure.
+
+**The fix**: Changed `_apply_moderation()` to:
+1. Still remove moderated-out participants (rows) — unchanged
+2. Zero out moderated-out comment columns instead of removing them
+3. `rating_mat` now has the same column count as `raw_rating_mat`
+
+**Impact on downstream**:
+- `tids` output now includes moderated-out tids (matching Clojure)
+- PCA: zeroed columns contribute nothing to variance, so PCA results are effectively identical
+- Repness: zeroed columns get na=0, nd=0, failing significance — effectively excluded
+- Vote counting: `user-vote-counts` in `to_math_blob()` uses `rating_mat`, so moderated
+  columns now count as "pass" votes (matching Clojure's behavior with zeroed columns)
+
+### Tests
+
+**New synthetic tests** (`TestD15SyntheticModeration`, 5 tests):
+- `test_zeroing_preserves_columns` — moderated columns still present
+- `test_zeroed_columns_are_all_zero` — moderated column values are 0.0
+- `test_non_moderated_columns_unchanged` — other columns retain original values
+- `test_empty_moderation_no_change` — no-op when no moderation
+- `test_moderate_nonexistent_tid` — graceful handling of unknown tids
+
+**Enhanced real-data tests** (`TestD15ModerationHandling`, 2 tests):
+- `test_moderated_comments_zeroed_not_removed` — applies mod-out from Clojure blob, checks column count and zeroed values
+- `test_tids_include_moderated` — verifies moderated tids remain in rating_mat columns
+
+**Updated existing tests**:
+- `test_conversation.py::test_moderation` — updated to expect zeroed columns
+- `test_conversation.py::test_update_moderation` — same
+- `test_discrepancy_fixes.py::TestD2cVoteCountSource::test_n_cmts_includes_moderated_out_comments` — updated comment count assertion
+
+### Test results
+
+- Public datasets: **328 passed, 0 failed, 6 skipped, 56 xfailed**
+- Private datasets: 13 failures — all **pre-existing** (golden snapshot staleness from earlier fixes, not D15-related). Verified by running parent branch.
+
+### What's next
+
+- D12 (comment priorities) or D1/D1b (PCA sign flips) — per plan ordering
+
+---
+
 ## Notes for Future Sessions
 
 - Private datasets are in `delphi/real_data/.local/` (separate git repo, linked via `link-to-polis-worktree.sh`)

delphi/docs/PLAN_DISCREPANCY_FIXES.md

@@ -47,6 +47,9 @@ Because this work will span multiple Claude Code sessions, we maintain:
 ### Testing Principles
 
 - **Granular tests per discrepancy**: Not just overall regression — each fix gets its own targeted test checking the specific aspect it addresses. Multiple discrepancies may affect `test_basic_outputs`; we need to see incremental improvement per fix.
+- **Clojure blob comparison is MANDATORY**: Every fix PR must include tests that compare Python output to actual Clojure math blob values — not just formula re-implementation tests (which are tautological: they only verify our code matches our reading of the Clojure, not that it matches Clojure's actual output). The Clojure blob is the ground truth oracle.
+- **Stage isolation via blob injection**: Since upstream stages (PCA, clustering) may not match between Python and Clojure, tests must inject Clojure blob values as inputs to the stage being tested, then compare outputs. For example: to test `prop_test` (D5), extract `n-success` and `n-trials` from the Clojure blob's `repness` entries, feed them to Python's `prop_test()`, and compare the result to the blob's `p-test`. This isolates each stage from upstream divergence.
+- **Blob fields available for injection/comparison**: The Clojure cold-start blob provides per-group repness entries with: `n-success` (=na), `n-trials` (=ns), `p-success` (=pa), `p-test` (=pat), `repness` (=ra), `repness-test` (=rat), `repful-for`, `best-agree`, `tid`. Also: `group-clusters` (memberships), `group-votes` (per-group vote counts), `consensus` (selected consensus comments), `comment-priorities` (per-tid priority values), `in-conv` (participant list).
 - **Targeted pipeline-stage tests**: For D2/D3 (participant filtering, clustering), check in-conv count, cluster count, and cluster memberships against Clojure blob. For D12, check comment-priorities against Clojure blob.
 - **All datasets, not just biodiversity**: Every fix must pass on ALL datasets. biodiversity is just one reference among many.
 - **Synthetic edge-case tests**: Every time we discover an edge case specific to one conversation, extract it into a synthetic unit test with made-up data (never real data from private datasets). These run fast and document the intent clearly.
@@ -441,6 +444,38 @@ By this point, we should have good test coverage from all the per-discrepancy te
 
 ---
 
+### Investigation: Cold-Start K Divergence (after D15, before D12)
+
+**Prerequisite**: All cold-start-relevant upstream fixes complete: D2/D2c/D2b (in-conv,
+vote counts, sort order), D15 (moderation handling). Note: D1 (PCA sign flips) only
+affects incremental updates — on cold start there are no previous components to align to.
+
+After D15, the rating matrix construction, in-conv filtering, and PCA inputs should all
+match Clojure. Both implementations use silhouette for k-selection. Yet on vw, Python
+selects k=4 while Clojure selects k=2.
+
+**Investigation steps**:
+
+1. **PCA component comparison**: Feed the same rating matrix to both sklearn TruncatedSVD
+   and a Python reimplementation of Clojure's power iteration. Quantify divergence
+   (cosine similarity per component, Frobenius norm).
+2. **Projection comparison**: Inject Clojure blob's PCA components into Python's
+   clustering path. Does k now match?
+3. **Base-cluster comparison**: Given the same projections, compare k-means centroids
+   and member assignments. Check initialization (Clojure uses first-k-distinct centers
+   from base clusters — does Python match?).
+4. **Silhouette score comparison**: Given the same base clusters, compare per-k
+   silhouette scores. Are the scores close but the winner differs?
+5. **All datasets**: Run on all datasets with cold-start blobs, not just vw.
+
+**Outcome**: Either (a) identify a fixable discrepancy that makes k match, or
+(b) document the inherent numerical divergence between sklearn SVD and Clojure
+power iteration, and establish tolerance bounds for k agreement in tests.
+
+See `delphi/docs/HANDOFF_K_DIVERGENCE_INVESTIGATION.md` for detailed context.
+
+---
+
 ### Explicitly Deferred
 
 - **D13 — Subgroup Clustering**: Not implemented in Python, never used by TypeScript consumers. No fix needed.
@@ -470,14 +505,15 @@ By this point, we should have good test coverage from all the per-discrepancy te
 | D5 | Proportion test | **PR 4** | — | **DONE** ✓ |
 | D6 | Two-proportion test | **PR 5** | — | **DONE** ✓ |
 | D7 | Repness metric | PR 6 | — | **DONE** ✓ |
-| D8 | Finalize cmt stats | PR 7 | — | Fix |
+| D8 | Finalize cmt stats | PR 7 | — | **DONE** ✓ |
 | D9 | Z-score thresholds | **PR 3** | **#2446** | **DONE** ✓ |
 | D10 | Rep comment selection | PR 8 | — | Fix (with legacy env var) |
 | D11 | Consensus selection | PR 9 | — | Fix (with legacy env var) |
 | D12 | Comment priorities | PR 11 | — | Fix (implement from scratch) |
 | D13 | Subgroup clustering | — | — | **Deferred** (unused) |
 | D14 | Large conv optimization | — | — | **Deferred** (Python fast enough) |
-| D15 | Moderation handling | PR 12 | — | Fix |
+| D15 | Moderation handling | PR 12 | — | **DONE** ✓ |
+| K-inv | Cold-start k divergence | (investigation) | — | Branch off D15 (D2+D15 done, clustering independent of repness) |
 | Replay | Replay infrastructure (A/B/C) | — | — | NOT BUILT — D3/D1 used synthetic tests only. Needed for incremental blob comparison. |
 
 ### Non-discrepancy PRs in the stack

delphi/polismath/conversation/conversation.py

@@ -297,15 +297,23 @@ def update_votes(self,
     def _apply_moderation(self) -> None:
         """
         Apply moderation settings to create filtered rating matrix.
+
+        Matches Clojure behavior (named_matrix.clj:214-230):
+        - Moderated-out participants are removed (rows dropped)
+        - Moderated-out comments are ZEROED OUT, not removed — the column
+          stays in the matrix with all values set to 0.  This preserves
+          matrix structure so that tids, column indices, and dimensions
+          match between Python and Clojure.
         """
-        # Filter out moderated participants and comments, and keep them sorted!
-        # Note: set operations are unordered, hence the extra sort.
-        # Natural sort: preserves types and sorts numerically when possible
+        # Filter out moderated participants (remove rows)
         keep_ptpts = natsorted(list(set(self.raw_rating_mat.index) - set(self.mod_out_ptpts)))
-        keep_comments = natsorted(list(set(self.raw_rating_mat.columns) - set(self.mod_out_tids)))
-        
-        # Create filtered matrix
-        self.rating_mat = self.raw_rating_mat.loc[keep_ptpts, keep_comments]
+        self.rating_mat = self.raw_rating_mat.loc[keep_ptpts].copy()
+
+        # Zero out moderated-out comments (keep columns, set values to 0)
+        # Clojure: (matrix/set-column m' i 0) — zeroes the column
+        mod_cols = [c for c in self.mod_out_tids if c in self.rating_mat.columns]
+        if mod_cols:
+            self.rating_mat[mod_cols] = 0.0
 
     def _compute_vote_stats(self) -> None:
         """

delphi/real_data/r4tykwac8thvzv35jrn53-biodiversity/golden_snapshot.json

@@ -8,7 +8,7 @@
     "n_comments_in_csv": 316,
     "n_participants_in_csv": 536,
     "fixed_timestamp": 1700000000000,
-    "recorded_at": "2026-03-24T11:45:32.247863"
+    "recorded_at": "2026-03-24T11:46:25.829282"
   },
   "stages": {
     "empty": {
@@ -46,7 +46,7 @@
         "disagree": [],
         "comment-stats": {}
       },
-      "math_tick": 27733
+      "math_tick": 27787
     },
     "after_load_no_compute": {
       "last_updated": 1700000000000,
@@ -8057,7 +8057,7 @@
         "disagree": [],
         "comment-stats": {}
       },
-      "math_tick": 27733
+      "math_tick": 27787
     },
     "after_pca": {
       "last_updated": 1700000000000,
@@ -19166,7 +19166,7 @@
         "disagree": [],
         "comment-stats": {}
       },
-      "math_tick": 27733
+      "math_tick": 27788
     },
     "after_clustering": {
       "last_updated": 1700000000000,
@@ -35107,7 +35107,7 @@
         "disagree": [],
         "comment-stats": {}
       },
-      "math_tick": 27733
+      "math_tick": 27788
     },
     "after_full_recompute": {
       "last_updated": 1700000000000,
@@ -61837,7 +61837,7 @@
         "disagree": [],
         "comment-stats": {}
       },
-      "math_tick": 27733
+      "math_tick": 27788
     },
     "full_data_export": {
       "conversation_id": "biodiversity",
@@ -81312,57 +81312,57 @@
   },
   "timing_stats": {
     "empty": {
-      "mean": 0.0002450136623034875,
-      "std": 4.3639502782565e-06,
+      "mean": 0.02019661133332799,
+      "std": 0.0345314761572571,
       "raw": [
-        0.0002470829931553453,
-        0.00023999999393709004,
-        0.000247957999818027
+        0.00025466700026299804,
+        0.0002650420065037906,
+        0.06007012499321718
       ]
     },
     "after_load_no_compute": {
-      "mean": 0.04123843066918198,
-      "std": 0.014000114032996457,
+      "mean": 0.048245472333898455,
+      "std": 0.01785033734259351,
       "raw": [
-        0.029898249995312653,
-        0.036930875008692965,
-        0.05688616700354032
+        0.030253250006353483,
+        0.04853270899911877,
+        0.06595045799622312
       ]
     },
     "after_pca": {
-      "mean": 0.02450374999898486,
-      "std": 0.01127007663956127,
+      "mean": 0.06067052799820279,
+      "std": 0.01914579351316182,
       "raw": [
-        0.013971791995572858,
-        0.03638962499098852,
-        0.023149833010393195
+        0.06774112499260809,
+        0.0752754170098342,
+        0.03899504199216608
       ]
     },
     "after_clustering": {
-      "mean": 0.17774104166892357,
-      "std": 0.11087459232976675,
+      "mean": 0.2687555973340447,
+      "std": 0.21711782082707126,
       "raw": [
-        0.07256712499656714,
-        0.16710608400171623,
-        0.29354991600848734
+        0.07044791700900532,
+        0.5007465409871656,
+        0.23507233400596306
       ]
     },
     "after_full_recompute": {
-      "mean": 0.27918515300067764,
-      "std": 0.10193353078517546,
+      "mean": 0.561707333336623,
+      "std": 0.15916801118706728,
       "raw": [
-        0.3940678340004524,
-        0.19956449999881443,
-        0.24392312500276603
+        0.39767841700813733,
+        0.7155225000024075,
+        0.5719210829993244
       ]
     },
     "full_data_export": {
-      "mean": 0.00023036100416599462,
-      "std": 4.601970059567263e-05,
+      "mean": 0.0025394716697822637,
+      "std": 0.0039485346647779385,
       "raw": [
-        0.0001994580088648945,
-        0.00028325000312179327,
-        0.0002083750005112961
+        0.0070985409984132275,
+        0.0002142080047633499,
+        0.0003056660061702132
       ]
     }
   }