feat(founder): add deterministic founder validation pipeline scaffold#23627
feat(founder): add deterministic founder validation pipeline scaffold#23627BrianCLong wants to merge 1 commit intomainfrom
Conversation
BrianCLong
added
the
codex
ChatGPT Codex Connector
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 19 minutes and 2 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughA new founder pipeline feature is introduced to validate founder ideas through deterministic evidence ID generation, problem definition validation, scoring, and metrics building, with accompanying test coverage and updated documentation. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Pipeline as runFounderPipeline()
participant Validator as scoreFounderIdea()
participant Schema as validateProblemDefinition()
participant Metrics as buildFounderMetrics()
participant Report as Report Builder
Client->>Pipeline: Input (title, summary, problem, evidence)
Pipeline->>Schema: validateProblemDefinition(problem)
Schema-->>Pipeline: checks {problemDefined, evidenceBacked, alternativesListed}
Pipeline->>Validator: scoreFounderIdea(input)
Validator->>Validator: buildFounderEvidenceId for each evidence item
Validator->>Validator: Calculate score = satisfiedChecks / 3
Validator-->>Pipeline: ValidationResult {score, passed, threshold, evidenceIds, checks}
Pipeline->>Metrics: buildFounderMetrics(input)
Metrics-->>Pipeline: FounderMetrics {score, passed, latency_budget_ms: 2000, ...}
Pipeline->>Report: Determine recommended_action
Report-->>Pipeline: "advance" if passed, "revise" otherwise
Pipeline-->>Client: FounderPipelineOutput {report, validation, metrics, stamp}
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79aecbc5d3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const checks = validateProblemDefinition(input.problem); | ||
| const evidenceIds = input.evidence.map((item) => buildFounderEvidenceId("validation", item)); | ||
|
|
||
| checks.evidenceBacked = evidenceIds.length > 0; |
There was a problem hiding this comment.
In scoreFounderIdea, checks.evidenceBacked is set using only evidenceIds.length > 0, so inputs like evidence: [" "] are treated as evidence-backed and can satisfy the gate even when no substantive evidence exists. Because this check contributes directly to passed, the pipeline can incorrectly advance low-quality ideas; filter/trim evidence entries before generating IDs and scoring.
Useful? React with 👍 / 👎.
| return { | ||
| problemDefined: Boolean(problem.customer.trim() && problem.pain_point.trim()), | ||
| evidenceBacked: true, | ||
| alternativesListed: problem.existing_alternatives.length > 0, |
There was a problem hiding this comment.
validateProblemDefinition marks alternativesListed true when existing_alternatives.length > 0, which accepts arrays containing only empty/whitespace strings (for example [" "]). This inflates the validation score and can flip outcomes to pass despite missing real alternative analysis; trim and validate at least one non-empty alternative value.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (5)
src/agents/founder/validators.ts (1)
15-16: Score denominator should be derived from active checks.Line 16 hardcodes
3; this becomes incorrect if checks change. Compute denominator fromObject.keys(checks).length.Proposed fix
- const satisfiedChecks = Object.values(checks).filter(Boolean).length; - const score = Number((satisfiedChecks / 3).toFixed(4)); + const totalChecks = Object.keys(checks).length; + const satisfiedChecks = Object.values(checks).filter(Boolean).length; + const score = totalChecks === 0 ? 0 : Number((satisfiedChecks / totalChecks).toFixed(4));🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/agents/founder/validators.ts` around lines 15 - 16, The score currently uses a hardcoded denominator (3); update the computation in the validators module so the denominator is derived from the active checks: use Object.keys(checks).length instead of 3 when computing score (the variables to change are satisfiedChecks and score where checks is referenced), and guard against division-by-zero (e.g., treat length 0 as 1 or return 0) so the score calculation remains correct if the set of checks changes.src/agents/founder/scoring.ts (1)
12-20: Avoid recomputing validation inside metrics builder.Line 13 recomputes
scoreFounderIdea, while the pipeline already computes validation. Pass validation intobuildFounderMetricsto prevent drift and duplicated logic.Proposed refactor
-import { FounderIdeaInput } from "./schemas"; -import { scoreFounderIdea } from "./validators"; +import { ValidationResult } from "./schemas"; export interface FounderMetrics { @@ -export function buildFounderMetrics(input: FounderIdeaInput): FounderMetrics { - const validation = scoreFounderIdea(input); - +export function buildFounderMetrics( + validation: Pick<ValidationResult, "score" | "passed"> +): FounderMetrics { return { latency_budget_ms: 2000, memory_budget_mb: 200, cost_budget: "negligible", score: validation.score, passed: validation.passed, }; }And in
src/agents/founder/pipeline.ts:- const metrics = buildFounderMetrics(input); + const metrics = buildFounderMetrics(validation);🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/agents/founder/scoring.ts` around lines 12 - 20, buildFounderMetrics currently recomputes validation by calling scoreFounderIdea(input); change the function signature to accept the already-computed validation object (e.g., pass a Validation or the result type returned by scoreFounderIdea) instead of FounderIdeaInput so buildFounderMetrics(validation) uses validation.score and validation.passed directly, remove the internal call to scoreFounderIdea, and update all call sites (e.g., in the pipeline) to pass the precomputed validation result into buildFounderMetrics to avoid duplicated logic and drift.src/agents/founder/schemas.ts (2)
35-40:validateProblemDefinitionshould not hardcodeevidenceBacked: true.Line 38 introduces a misleading default for a check that depends on
input.evidence, not problem definition. Keep this function scoped to problem-only checks and composeevidenceBackedinvalidators.ts.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/agents/founder/schemas.ts` around lines 35 - 40, The validateProblemDefinition function currently hardcodes evidenceBacked: true; remove that key from validateProblemDefinition so it returns only problem-only checks (e.g., problemDefined and alternativesListed) for the ProblemDefinition shape, and then in validators.ts compose the complete ValidationResult["checks"] by adding evidenceBacked based on input.evidence (e.g., Boolean(input.evidence && input.evidence.trim())). Update any callers that expect evidenceBacked from validateProblemDefinition to instead read the composed checks from validators.ts.
30-33: Constrain evidencestageto known literals.Line 30 accepts arbitrary strings, which can drift from the intended ID scheme. A literal union helps keep IDs governance-safe.
Proposed refactor
+export type FounderEvidenceStage = "problem" | "validation" | "report" | "metrics" | "stamp"; - -export function buildFounderEvidenceId(stage: string, payload: string): string { +export function buildFounderEvidenceId(stage: FounderEvidenceStage, payload: string): string { const hash = createHash("sha256").update(payload).digest("hex").slice(0, 10); return `EVID-FOUND-${stage}-${hash}`; }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/agents/founder/schemas.ts` around lines 30 - 33, The buildFounderEvidenceId function allows any string for the stage parameter which can drift; restrict stage to an explicit string literal union (e.g., export type FounderEvidenceStage = 'proposal' | 'review' | 'approved' | 'deployed' — or the actual set your system uses) and change the signature to buildFounderEvidenceId(stage: FounderEvidenceStage, payload: string): string; update any callers to use the new FounderEvidenceStage type (or cast only where appropriate) so IDs remain constrained and type-checked across the codebase.src/agents/founder/__tests__/schemas.test.ts (1)
15-32: Add an assertion for the default feature-flag OFF behavior.This suite currently checks deterministic output but doesn’t verify the stated default
founder_pipeline_v1: falsebehavior.Proposed test addition
it("produces stable report artifacts and enforces threshold", () => { const output = runFounderPipeline({ @@ expect(output.stamp.deterministic).toBe(true); + expect(output.stamp.enabled).toBe(false); expect(output.validation.threshold).toBe(FOUNDER_VALIDATION_THRESHOLD);🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/agents/founder/__tests__/schemas.test.ts` around lines 15 - 32, The test must assert the default feature-flag is OFF: add an assertion after runFounderPipeline to verify that the `founder_pipeline_v1` flag is false (e.g. assert via whatever feature-flag API your code uses or via flags exposed on the pipeline output). Specifically, in the same test that calls `runFounderPipeline(...)` and checks `output.stamp` and `output.validation` (and uses `FOUNDER_VALIDATION_THRESHOLD`), add `expect(...)` to confirm `founder_pipeline_v1` is false (for example `expect(getFeatureFlag('founder_pipeline_v1')).toBe(false)` or `expect(output.features?.founder_pipeline_v1).toBe(false)` depending on where flags are surfaced).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@repo_assumptions.md`:
- Around line 8-10: The three bullets claiming absence of a "founder
intelligence" pipeline, a structured business validation schema, and a
lightweight financial modeling module are now stale because this PR added the
founder scaffold under src/agents/founder; update the repo_assumptions.md
entries to reflect that a founder pipeline and associated modules exist (mention
src/agents/founder, the founder intelligence pipeline, the business validation
schema, and the financial modeling module) or change the bullets to note their
current status (implemented/partial/planned) to avoid misleading follow-ups.
- Line 19: The checklist currently points to src/graphrag but the founder
validation schemas live in the founder schemas file; update the checklist entry
to reference the correct schema location by replacing the target with schemas.ts
in agents/founder (the founder validation schema module) so the checklist points
to the actual implementation used for founder validation.
In `@src/agents/founder/pipeline.ts`:
- Around line 35-43: The recommended_action can be "advance" even when the
feature flag (stamp.feature_flag / stamp.enabled) is false; update the logic
that builds recommended_action so it only yields "advance" when
validation.passed AND the feature flag is enabled (use stamp.enabled or
config.founder_pipeline_v1), otherwise return the non-flagged fallback (e.g.,
"revise" or the prior behavior). Locate the object that sets recommended_action
and change the ternary/assignment to include the feature-flag check alongside
validation.passed (reference recommended_action, validation.passed,
stamp.enabled, and config.founder_pipeline_v1).
In `@src/agents/founder/validators.ts`:
- Around line 11-13: The code currently treats any entry in input.evidence as
valid (so whitespace-only strings pass); before mapping to IDs with
buildFounderEvidenceId and before setting checks.evidenceBacked, normalize and
filter out falsy/whitespace-only entries by trimming each item (e.g., const
cleaned = input.evidence.map(s => s?.trim()).filter(Boolean)), then map cleaned
to evidenceIds via buildFounderEvidenceId and set checks.evidenceBacked =
evidenceIds.length > 0 so only non-empty evidence counts.
---
Nitpick comments:
In `@src/agents/founder/__tests__/schemas.test.ts`:
- Around line 15-32: The test must assert the default feature-flag is OFF: add
an assertion after runFounderPipeline to verify that the `founder_pipeline_v1`
flag is false (e.g. assert via whatever feature-flag API your code uses or via
flags exposed on the pipeline output). Specifically, in the same test that calls
`runFounderPipeline(...)` and checks `output.stamp` and `output.validation` (and
uses `FOUNDER_VALIDATION_THRESHOLD`), add `expect(...)` to confirm
`founder_pipeline_v1` is false (for example
`expect(getFeatureFlag('founder_pipeline_v1')).toBe(false)` or
`expect(output.features?.founder_pipeline_v1).toBe(false)` depending on where
flags are surfaced).
In `@src/agents/founder/schemas.ts`:
- Around line 35-40: The validateProblemDefinition function currently hardcodes
evidenceBacked: true; remove that key from validateProblemDefinition so it
returns only problem-only checks (e.g., problemDefined and alternativesListed)
for the ProblemDefinition shape, and then in validators.ts compose the complete
ValidationResult["checks"] by adding evidenceBacked based on input.evidence
(e.g., Boolean(input.evidence && input.evidence.trim())). Update any callers
that expect evidenceBacked from validateProblemDefinition to instead read the
composed checks from validators.ts.
- Around line 30-33: The buildFounderEvidenceId function allows any string for
the stage parameter which can drift; restrict stage to an explicit string
literal union (e.g., export type FounderEvidenceStage = 'proposal' | 'review' |
'approved' | 'deployed' — or the actual set your system uses) and change the
signature to buildFounderEvidenceId(stage: FounderEvidenceStage, payload:
string): string; update any callers to use the new FounderEvidenceStage type (or
cast only where appropriate) so IDs remain constrained and type-checked across
the codebase.
In `@src/agents/founder/scoring.ts`:
- Around line 12-20: buildFounderMetrics currently recomputes validation by
calling scoreFounderIdea(input); change the function signature to accept the
already-computed validation object (e.g., pass a Validation or the result type
returned by scoreFounderIdea) instead of FounderIdeaInput so
buildFounderMetrics(validation) uses validation.score and validation.passed
directly, remove the internal call to scoreFounderIdea, and update all call
sites (e.g., in the pipeline) to pass the precomputed validation result into
buildFounderMetrics to avoid duplicated logic and drift.
In `@src/agents/founder/validators.ts`:
- Around line 15-16: The score currently uses a hardcoded denominator (3);
update the computation in the validators module so the denominator is derived
from the active checks: use Object.keys(checks).length instead of 3 when
computing score (the variables to change are satisfiedChecks and score where
checks is referenced), and guard against division-by-zero (e.g., treat length 0
as 1 or return 0) so the score calculation remains correct if the set of checks
changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 8a01c790-b0ae-46bb-81ab-6ba90d563f4d
📒 Files selected for processing (6)
repo_assumptions.mdsrc/agents/founder/__tests__/schemas.test.tssrc/agents/founder/pipeline.tssrc/agents/founder/schemas.tssrc/agents/founder/scoring.tssrc/agents/founder/validators.ts
| - No existing "founder intelligence" pipeline under `src/agents`. | ||
| - No structured business validation schema for founder idea intake. | ||
| - No lightweight financial modeling module specific to founder pipeline validation. |
There was a problem hiding this comment.
Stale assumptions now conflict with merged founder scaffold.
Line 8–Line 10 state these modules do not exist, but this PR introduces them under src/agents/founder. Please update these bullets to avoid misleading follow-up work.
🧰 Tools
🪛 LanguageTool
[style] ~10-~10: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...ation schema for founder idea intake. - No lightweight financial modeling module s...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@repo_assumptions.md` around lines 8 - 10, The three bullets claiming absence
of a "founder intelligence" pipeline, a structured business validation schema,
and a lightweight financial modeling module are now stale because this PR added
the founder scaffold under src/agents/founder; update the repo_assumptions.md
entries to reflect that a founder pipeline and associated modules exist (mention
src/agents/founder, the founder intelligence pipeline, the business validation
schema, and the financial modeling module) or change the bullets to note their
current status (implemented/partial/planned) to avoid misleading follow-ups.
|
|
||
| ## Pre-PR Recommendation | ||
| Proceed with a minimal, feature-flagged implementation under `src/graphrag/policy/` first, with deterministic artifacts and schema validation wired to current evidence conventions. | ||
| - Confirm schema conventions in `src/graphrag`. |
There was a problem hiding this comment.
Checklist points to the wrong schema area for this feature.
Line 19 references src/graphrag, but founder validation schema conventions are implemented in src/agents/founder/schemas.ts. Consider updating the checklist target.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@repo_assumptions.md` at line 19, The checklist currently points to
src/graphrag but the founder validation schemas live in the founder schemas
file; update the checklist entry to reference the correct schema location by
replacing the target with schemas.ts in agents/founder (the founder validation
schema module) so the checklist points to the actual implementation used for
founder validation.
| recommended_action: validation.passed ? "advance" : "revise", | ||
| }, | ||
| validation, | ||
| metrics, | ||
| stamp: { | ||
| deterministic: true, | ||
| feature_flag: "founder_pipeline_v1", | ||
| enabled: config.founder_pipeline_v1, | ||
| }, |
There was a problem hiding this comment.
Feature flag does not currently gate decision output.
Line 35 can return "advance" even when Line 42 reports the feature as disabled. If this flag is meant to gate rollout, decision output should also be gated.
Proposed fix
export function runFounderPipeline(
input: FounderIdeaInput,
config: FounderPipelineConfig = { founder_pipeline_v1: false }
): FounderPipelineOutput {
+ const enabled = config.founder_pipeline_v1;
const validation = scoreFounderIdea(input);
const metrics = buildFounderMetrics(input);
return {
@@
- recommended_action: validation.passed ? "advance" : "revise",
+ recommended_action: enabled && validation.passed ? "advance" : "revise",
@@
- enabled: config.founder_pipeline_v1,
+ enabled,
},
};
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/agents/founder/pipeline.ts` around lines 35 - 43, The recommended_action
can be "advance" even when the feature flag (stamp.feature_flag / stamp.enabled)
is false; update the logic that builds recommended_action so it only yields
"advance" when validation.passed AND the feature flag is enabled (use
stamp.enabled or config.founder_pipeline_v1), otherwise return the non-flagged
fallback (e.g., "revise" or the prior behavior). Locate the object that sets
recommended_action and change the ternary/assignment to include the feature-flag
check alongside validation.passed (reference recommended_action,
validation.passed, stamp.enabled, and config.founder_pipeline_v1).
| const evidenceIds = input.evidence.map((item) => buildFounderEvidenceId("validation", item)); | ||
|
|
||
| checks.evidenceBacked = evidenceIds.length > 0; |
There was a problem hiding this comment.
Whitespace-only evidence can incorrectly pass validation.
Line 13 treats any array entry as evidence, including blank strings. Normalize and filter evidence before generating IDs/checking evidenceBacked.
Proposed fix
- const evidenceIds = input.evidence.map((item) => buildFounderEvidenceId("validation", item));
-
- checks.evidenceBacked = evidenceIds.length > 0;
+ const normalizedEvidence = input.evidence
+ .map((item) => item.trim())
+ .filter((item) => item.length > 0);
+ const evidenceIds = normalizedEvidence.map((item) =>
+ buildFounderEvidenceId("validation", item)
+ );
+
+ checks.evidenceBacked = normalizedEvidence.length > 0;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const evidenceIds = input.evidence.map((item) => buildFounderEvidenceId("validation", item)); | |
| checks.evidenceBacked = evidenceIds.length > 0; | |
| const normalizedEvidence = input.evidence | |
| .map((item) => item.trim()) | |
| .filter((item) => item.length > 0); | |
| const evidenceIds = normalizedEvidence.map((item) => | |
| buildFounderEvidenceId("validation", item) | |
| ); | |
| checks.evidenceBacked = normalizedEvidence.length > 0; |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/agents/founder/validators.ts` around lines 11 - 13, The code currently
treats any entry in input.evidence as valid (so whitespace-only strings pass);
before mapping to IDs with buildFounderEvidenceId and before setting
checks.evidenceBacked, normalize and filter out falsy/whitespace-only entries by
trimming each item (e.g., const cleaned = input.evidence.map(s =>
s?.trim()).filter(Boolean)), then map cleaned to evidenceIds via
buildFounderEvidenceId and set checks.evidenceBacked = evidenceIds.length > 0 so
only non-empty evidence counts.
… (rebase on main)
BrianCLong
force-pushed
the
codex/create-founder-pipeline-for-business-validation
branch
from
79aecbc to
6bd02b2
Compare
1 participant
Motivation
founder_pipeline_v1defaulting to OFF to avoid affecting existing workflows.Description
src/agents/founderscaffold includingschemas.ts(typed contracts,buildFounderEvidenceId,FOUNDER_VALIDATION_THRESHOLD),validators.ts(validation & scoring),scoring.ts(performance/cost budgets + metric builder), andpipeline.ts(orchestration and deterministic output shape).EVID-FOUND-<stage>-<hash>using SHA-256 to produce stable, deterministic IDs for CI traceability.0.7and returnsreport,validation,metrics, andstampartifacts suitable forreport.json,validation.json,metrics.json, andstamp.jsonoutputs.src/agents/founder/__tests__/schemas.test.tsand updaterepo_assumptions.mddocumenting verified/assumed boundaries and a validation checklist.Testing
pnpm exec prettier --checkandpnpm exec prettier --writeon the new files, after which all files matched Prettier style (check passed).pnpm exec vitest run src/agents/founder/__tests__/schemas.test.ts, and the test suite passed (1 file, 2 tests, all passed).artifacts/founder-plan/(report.json, validation.json, metrics.json, stamp.json) during the rollout to demonstrate deterministic outputs and expected shapes.Codex Task
Summary by CodeRabbit
Release Notes
New Features
Tests
Documentation