[Stack 20/27] Fix D15: match Clojure moderation handling (zero out columns, don't remove) by jucor · Pull Request #2452 · compdemocracy/polis

jucor · 2026-03-17T18:35:22Z

Summary

Stacked on #2451 (Fix D8: match Clojure repful classification (rat > rdt)). Please review and merge #2451 first.
Next in stack: #2453 (Fix K-means k divergence: preserve vote-encounter row order)

Python's _apply_moderation() removed moderated-out comment columns entirely
from rating_mat. Clojure's zero-out-columns (named_matrix.clj:214-230) sets
all values in moderated columns to 0, preserving matrix structure.

This fix changes Python to match:

Moderated-out comment columns are zeroed (values set to 0.0), not removed
rating_mat retains the same column count as raw_rating_mat
Moderated-out participants (rows) are still removed — unchanged

Why zeroing matters

Matrix dimensions: Clojure's rating-mat has the same shape as raw-rating-mat.
Downstream code (PCA, repness) processes the same-shaped matrix.
tids list: Column indices stay stable. Consumers depend on this.
Practical impact: Zeroed columns have no signal (na=0, nd=0), so they fail
significance tests and are excluded from repness/consensus. PCA sees zero variance.

Changes

conversation.py: _apply_moderation() — zero out columns instead of removing
test_discrepancy_fixes.py: 5 new synthetic tests + 2 enhanced real-data tests
test_conversation.py: Updated to expect zeroed columns

Test plan

Synthetic tests: zeroing preserves columns, values are 0, non-moderated unchanged
Real-data test: biodiversity-incremental (169 mod-out comments)
Full public test suite: 328 passed, 0 failed
TDD cycle: RED (2 failures) → GREEN (all pass)

🤖 Generated with Claude Code

Copilot

Pull request overview

Aligns Python’s moderation handling with Clojure’s behavior by preserving the vote-matrix column structure when comments are moderated out (zeroing moderated columns rather than removing them), ensuring downstream math (PCA/repness) sees consistent dimensions/column indices.

Changes:

Update _apply_moderation() to drop moderated-out participants (rows) but zero-out moderated-out comment columns (instead of removing them).
Update/add tests to expect preserved columns and validate moderated columns are zeroed.
Mark D15 as complete in discrepancy tracking docs/journal.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`delphi/polismath/conversation/conversation.py`	Change moderation application to zero moderated-out columns while preserving matrix shape.
`delphi/tests/test_discrepancy_fixes.py`	Update D2c expectation for column counts; add/extend D15 parity + synthetic moderation tests.
`delphi/tests/test_conversation.py`	Update moderation-related assertions to expect kept-but-zeroed comment columns.
`delphi/docs/PLAN_DISCREPANCY_FIXES.md`	Mark D15 as DONE and adjust next-step notes.
`delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`	Add journal entry documenting D15 work and update “What’s Next”.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

delphi/polismath/conversation/conversation.py

+
+        # Zero out moderated-out comments (keep columns, set values to 0)
+        # Clojure: (matrix/set-column m' i 0) — zeroes the column
+        mod_cols = [c for c in self.mod_out_tids if c in self.rating_mat.columns]


delphi/tests/test_discrepancy_fixes.py

+        # Moderated columns should be all zeros (not NaN, not original values)
+        for tid in mod_out:
+            if tid in mod_conv.rating_mat.columns:
+                col_values = mod_conv.rating_mat[tid].values
+                check.is_true(
+                    np.all(col_values == 0.0),
+                    f"Moderated tid {tid} should be all zeros, "
+                    f"got non-zero values: {col_values[col_values != 0.0][:5]}"
+                )
+
+    def test_tids_include_moderated(self, conv, clojure_blob, dataset_name):
+        """The tids output should include moderated-out comments (matching Clojure)."""
+        mod_out = clojure_blob.get('mod-out') or []
+        if not mod_out:
+            pytest.skip(f"[{dataset_name}] No moderated comments in this dataset")
+
+        mod_conv = conv.update_moderation(
+            {'mod_out_tids': mod_out},
+            recompute=False,
+        )
+
+        # rating_mat.columns (used for tids output) should include moderated tids
+        for tid in mod_out:
+            if tid in mod_conv.raw_rating_mat.columns:
+                check.is_in(
+                    tid, set(mod_conv.rating_mat.columns),


delphi/tests/test_discrepancy_fixes.py

+        n_cols_raw = len(mod_conv.raw_rating_mat.columns)
        n_tids_clojure = len(clojure_blob.get('tids', []))

-        print(f"[{dataset_name}] Moderated comments: {len(mod_out)}")
-        print(f"[{dataset_name}] Python matrix columns: {n_cols_python}, Clojure tids: {n_tids_clojure}")
+        print(f"[{dataset_name}] mod-out: {len(mod_out)}")
+        print(f"[{dataset_name}] Python rating_mat cols: {n_cols_python}, raw cols: {n_cols_raw}")
+        print(f"[{dataset_name}] Clojure tids: {n_tids_clojure}")


github-actions · 2026-03-30T13:44:37Z

Delphi Coverage Report

File	Stmts	Miss	Cover
init.py	2	0	100%
benchmarks/bench_pca.py	76	76	0%
benchmarks/bench_repness.py	81	81	0%
benchmarks/bench_update_votes.py	38	38	0%
benchmarks/benchmark_utils.py	34	34	0%
components/init.py	1	0	100%
components/config.py	165	133	19%
conversation/init.py	2	0	100%
conversation/conversation.py	1109	320	71%
conversation/manager.py	131	42	68%
database/init.py	1	0	100%
database/dynamodb.py	387	234	40%
database/postgres.py	305	205	33%
pca_kmeans_rep/init.py	5	0	100%
pca_kmeans_rep/clusters.py	257	22	91%
pca_kmeans_rep/corr.py	98	17	83%
pca_kmeans_rep/pca.py	52	16	69%
pca_kmeans_rep/repness.py	312	34	89%
regression/init.py	4	0	100%
regression/clojure_comparer.py	188	17	91%
regression/comparer.py	887	720	19%
regression/datasets.py	135	27	80%
regression/recorder.py	36	27	25%
regression/utils.py	138	87	37%
run_math_pipeline.py	260	114	56%
umap_narrative/500_generate_embedding_umap_cluster.py	210	109	48%
umap_narrative/501_calculate_comment_extremity.py	112	53	53%
umap_narrative/502_calculate_priorities.py	135	135	0%
umap_narrative/700_datamapplot_for_layer.py	502	502	0%
umap_narrative/701_static_datamapplot_for_layer.py	310	310	0%
umap_narrative/702_consensus_divisive_datamapplot.py	432	432	0%
umap_narrative/801_narrative_report_batch.py	785	785	0%
umap_narrative/802_process_batch_results.py	265	265	0%
umap_narrative/803_check_batch_status.py	175	175	0%
umap_narrative/llm_factory_constructor/init.py	2	2	0%
umap_narrative/llm_factory_constructor/model_provider.py	157	157	0%
umap_narrative/polismath_commentgraph/init.py	1	0	100%
umap_narrative/polismath_commentgraph/cli.py	270	270	0%
umap_narrative/polismath_commentgraph/core/init.py	3	3	0%
umap_narrative/polismath_commentgraph/core/clustering.py	108	108	0%
umap_narrative/polismath_commentgraph/core/embedding.py	104	104	0%
umap_narrative/polismath_commentgraph/lambda_handler.py	219	219	0%
umap_narrative/polismath_commentgraph/schemas/init.py	2	0	100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py	160	9	94%
umap_narrative/polismath_commentgraph/tests/conftest.py	17	17	0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py	74	74	0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py	55	55	0%
umap_narrative/polismath_commentgraph/tests/test_storage.py	87	87	0%
umap_narrative/polismath_commentgraph/utils/init.py	3	0	100%
umap_narrative/polismath_commentgraph/utils/converter.py	283	237	16%
umap_narrative/polismath_commentgraph/utils/group_data.py	354	336	5%
umap_narrative/polismath_commentgraph/utils/storage.py	584	518	11%
umap_narrative/reset_conversation.py	159	50	69%
umap_narrative/run_pipeline.py	453	312	31%
utils/general.py	62	41	34%
Total	10787	7609	29%

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…emove) Python's _apply_moderation() removed moderated-out comment columns entirely from rating_mat. Clojure's zero-out-columns (named_matrix.clj:214-230) sets all values in moderated columns to 0, preserving matrix structure. Change _apply_moderation() to zero out moderated columns instead of removing them, so that: - rating_mat retains the same column count as raw_rating_mat - tids output includes moderated-out tids (matching Clojure) - Matrix dimensions are preserved through the pipeline Moderated-out participants (rows) are still removed — unchanged. Zeroed columns have no signal (na=0, nd=0), so they fail all significance tests and are effectively excluded from repness/consensus/PCA, but their presence preserves index stability for downstream consumers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

jucor · 2026-03-30T22:54:49Z

Superseded by spr-managed PR stack. See the new stack starting at #2508.

jucor requested a review from Copilot March 17, 2026 18:44

Copilot started reviewing on behalf of jucor March 17, 2026 18:45 View session

jucor changed the title ~~Fix D15: match Clojure moderation handling (zero out columns, don't remove)~~ [Stack 18/24] Fix D15: match Clojure moderation handling (zero out columns, don't remove) Mar 17, 2026

Copilot AI reviewed Mar 17, 2026

View reviewed changes

jucor marked this pull request as ready for review March 17, 2026 18:53

jucor changed the title ~~[Stack 18/24] Fix D15: match Clojure moderation handling (zero out columns, don't remove)~~ [Stack 18/25] Fix D15: match Clojure moderation handling (zero out columns, don't remove) Mar 17, 2026

This was referenced Mar 17, 2026

[Stack 19/27] Fix D8: match Clojure repful classification (rat > rdt) #2451

Closed

[Stack 21/27] Fix K-means k divergence: preserve vote-encounter row order #2453

Closed

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from d4e2154 to decac1a Compare March 17, 2026 20:51

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from 39fc7ef to c0684d4 Compare March 18, 2026 19:02

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from decac1a to 7611c85 Compare March 18, 2026 19:02

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from c0684d4 to 42d9f25 Compare March 18, 2026 19:07

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 7611c85 to f51d33f Compare March 18, 2026 19:13

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from 42d9f25 to b68c7e5 Compare March 19, 2026 10:23

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from f51d33f to 19a64ef Compare March 19, 2026 10:23

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from b68c7e5 to c2e521d Compare March 19, 2026 10:46

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 19a64ef to 29283da Compare March 19, 2026 10:49

jucor changed the title ~~[Stack 18/25] Fix D15: match Clojure moderation handling (zero out columns, don't remove)~~ [Stack 17/24] Fix D15: match Clojure moderation handling (zero out columns, don't remove) Mar 19, 2026

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from c2e521d to b8a8e08 Compare March 19, 2026 12:32

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 29283da to 9149555 Compare March 19, 2026 12:33

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from b8a8e08 to 0154ce7 Compare March 19, 2026 14:52

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 9149555 to 2b015fd Compare March 19, 2026 14:52

jucor changed the title ~~[Stack 17/24] Fix D15: match Clojure moderation handling (zero out columns, don't remove)~~ [Stack 18/25] Fix D15: match Clojure moderation handling (zero out columns, don't remove) Mar 19, 2026

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from 0154ce7 to a313f1c Compare March 23, 2026 15:33

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 2b015fd to 176da37 Compare March 23, 2026 15:34

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from a313f1c to 6a2ac55 Compare March 23, 2026 15:41

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 176da37 to 13fedd6 Compare March 23, 2026 15:41

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from 6a2ac55 to 90dd355 Compare March 23, 2026 15:44

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 13fedd6 to 39928d2 Compare March 23, 2026 15:44

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from 90dd355 to 1d05e99 Compare March 23, 2026 15:55

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 4e68f68 to 7313bc0 Compare March 24, 2026 11:35

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from 4ebd5ab to baceacd Compare March 24, 2026 11:45

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 7313bc0 to d8d3f24 Compare March 24, 2026 11:46

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from baceacd to 74b31de Compare March 26, 2026 21:24

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from d8d3f24 to 2ce0b36 Compare March 26, 2026 21:24

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from 74b31de to c4f5811 Compare March 27, 2026 01:15

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 2ce0b36 to 19e36d8 Compare March 27, 2026 01:15

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from c4f5811 to 2960412 Compare March 27, 2026 01:53

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch 2 times, most recently from 42a795a to 9bd7794 Compare March 27, 2026 02:10

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch 2 times, most recently from 44b04ae to b24d69b Compare March 27, 2026 10:41

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 9bd7794 to 04a595b Compare March 27, 2026 10:41

jucor changed the title ~~[Stack 18/25] Fix D15: match Clojure moderation handling (zero out columns, don't remove)~~ [Stack 19/26] Fix D15: match Clojure moderation handling (zero out columns, don't remove) Mar 30, 2026

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 04a595b to f7102c6 Compare March 30, 2026 12:48

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from b24d69b to abfbacb Compare March 30, 2026 12:48

jucor changed the title ~~[Stack 19/26] Fix D15: match Clojure moderation handling (zero out columns, don't remove)~~ [Stack 20/27] Fix D15: match Clojure moderation handling (zero out columns, don't remove) Mar 30, 2026

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from f7102c6 to 4e5f139 Compare March 30, 2026 12:54

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from abfbacb to 3847f76 Compare March 30, 2026 12:54

jucor requested a review from Copilot March 30, 2026 16:25

Copilot started reviewing on behalf of jucor March 30, 2026 16:26 View session

Copilot AI reviewed Mar 30, 2026

View reviewed changes

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from 3847f76 to f7062f8 Compare March 30, 2026 16:49

jucor force-pushed the jc/clj-parity-d15-moderation-handling-zeros-vs-removes branch from 4e5f139 to 20c419f Compare March 30, 2026 16:49

jucor force-pushed the jc/clj-parity-d8-finalize-stats branch from f7062f8 to f7329b4 Compare March 30, 2026 17:05

This was referenced Mar 30, 2026

IGNORE -- crash from spr #2504

Closed

IGNORE -- crash from spr #2506

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Stack 20/27] Fix D15: match Clojure moderation handling (zero out columns, don't remove)#2452

[Stack 20/27] Fix D15: match Clojure moderation handling (zero out columns, don't remove)#2452
jucor wants to merge 1 commit intojc/clj-parity-d8-finalize-statsfrom
jc/clj-parity-d15-moderation-handling-zeros-vs-removes

jucor commented Mar 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions bot commented Mar 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

jucor commented Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jucor commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why zeroing matters

Changes

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions bot commented Mar 30, 2026

Delphi Coverage Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

jucor commented Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jucor commented Mar 17, 2026 •

edited

Loading