Skip to content

feat(prompts): fix all CI/CD even when inherited under --auto-restart-until-mergeable (#1887)#1888

Open
konard wants to merge 5 commits into
mainfrom
issue-1887-778f06fa5585
Open

feat(prompts): fix all CI/CD even when inherited under --auto-restart-until-mergeable (#1887)#1888
konard wants to merge 5 commits into
mainfrom
issue-1887-778f06fa5585

Conversation

@konard

@konard konard commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Under --auto-restart-until-mergeable (also implied by --auto-merge), the solver restarts the AI session until the pull request is mergeable. If the AI decides a failing CI/CD check is pre-existing or inherited from another branch and merely asks a human for a decision instead of fixing it, the loop cannot converge without a human and keeps restarting — burning iterations and money (observed in lefinepro/kefine#170, $16+ by iteration 2).

This PR raises the probability the AI fixes the checks itself — without forcing it — by adding soft When x, do y. guidance in the auto-restart prompt and the system prompt (the issue explicitly allowed "or both").

Changes

  • src/solve.restart-shared.lib.mjsbuildAutoRestartInstructions() (the single shared auto-restart/watch feedback prompt used by every tool) now:
    • says to fix any failing check even if it looks pre-existing / inherited / unrelated,
    • explains that the session auto-restarts until mergeable, so leaving a check unaddressed loops indefinitely,
    • frames repository-wide breakage as in scope unless the scope is explicitly restricted,
    • still permits attempting the best fix first and then leaving a clear comment if a human decision is genuinely required.
  • src/{claude,codex,gemini,qwen,agent,opencode}.prompts.lib.mjsbuildSystemPrompt() for all six tools gains two When x, do y. statements (fix failing CI/CD even when pre-existing/inherited; keep the default branch clean by assuming repository-wide scope unless the user restricts it). Applied to all tools per the "fix in all places" requirement.

Why no hard gate

The issue asked to increase probability … but no forcing. A hard rule ("never end while CI is red") was deliberately rejected: genuine human-only blockers exist (infra, secrets, policy), and forcing would turn "loop asking" into "loop unable to stop". The change shifts the agent's default toward fixing while preserving the human-escalation escape hatch. The existing 5-iteration cap remains the runaway backstop.

How to reproduce the original problem

  1. Run solve on an issue with --auto-merge / --auto-restart-until-mergeable against a base branch whose CI is already red.
  2. The AI proves the failure is inherited and asks a human to decide.
  3. With no human reply, the auto-restart fires again with identical state → non-converging loop until the iteration limit.

Tests

  • tests/test-issue-1887-ci-fix-prompt.mjs — 16 assertions verifying the new guidance is present in the auto-restart prompt and in all six tools' system prompts.
  • Full suite: 237 test files pass, ESLint clean on src/, Prettier clean.

Case study (Requirements R4–R5)

Compiled under docs/case-studies/issue-1887/:

  • README.md — executive summary, reconstructed timeline, root cause, the fix.
  • requirements.md — R1–R9 mapped to how this PR addresses each.
  • analysis.md — restart-loop walkthrough, why the AI escalated, root-cause statement, why-not-hard-gate, verbose-sufficiency (R6 n/a), third-party reporting (R7 n/a).
  • solutions.md — options A–D considered and why A was chosen, plus follow-ups.
  • existing-components.md — in-repo and ecosystem prior art reused.
  • data/ — saved kefine issue/PR JSON, comment timeline, the verbatim before-fix prompt, and the AI's decision comments.

Closes #1887

Adding .gitkeep for PR creation (default mode).
This file will be removed when the task is complete.

Issue: #1887
@konard konard self-assigned this Jun 10, 2026
konard added 3 commits June 10, 2026 22:14
…ited (#1887)

When --auto-restart-until-mergeable is enabled, a PR whose CI/CD stays red
restarts forever. The AI must actually attempt to fix failing checks — even
when the breakage looks pre-existing or inherited from another branch —
instead of only reporting it and asking for a human decision.

- Add explicit fix-all-CI guidance + infinite-loop warning to
  buildAutoRestartInstructions() (shared by all auto-restart/watch iterations)
- Add two 'When x, do y.' statements to buildSystemPrompt() for all 6 AI tools
  (claude, codex, gemini, qwen, agent, opencode): fix failing CI/CD even if
  pre-existing, and keep the default branch clean by assuming repository-wide
  fix scope unless the user explicitly restricts it
- Add tests/test-issue-1887-ci-fix-prompt.mjs (16 assertions)
Compile the deep case study under docs/case-studies/issue-1887/ (README,
requirements, analysis, solutions, existing-components) with saved kefine
issue/PR data, the verbatim before-fix auto-restart prompt, and the AI's
decision comments. Add a minor changeset for the prompt change (#1887).
@konard konard changed the title [WIP] We need update system prompt to increase probability of AI models to actually fix all CI/CD in this pull request no matter if it was broken something else when --auto-restart-until-mergeable is enabled feat(prompts): fix all CI/CD even when inherited under --auto-restart-until-mergeable (#1887) Jun 10, 2026
@konard
konard marked this pull request as ready for review June 10, 2026 22:48
@konard

konard commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

Working session summary

Всё готово — это была фоновая задача ожидания CI, которая подтвердила, что все проверки прошли успешно.

Работа над issue #1887 полностью завершена. Итог:

Что сделано в PR #1888 (#1888):

  1. Корневая причина — это дефект коммуникации области видимости в промптах, а не баг в логике. Промпты узко формулировали требования к CI («comply with… and they pass») и нигде не объявляли, что унаследованные/общерепозиторные поломки находятся в области исправления, и не сообщали модели, что её сессия перезапускается до состояния mergeable.

  2. Исправление (мягкие подсказки, без принуждения — как требовал issue):

    • src/solve.restart-shared.lib.mjsbuildAutoRestartInstructions() (общий для всех инструментов и watch-режима);
    • системные промпты всех шести инструментов (claude, codex, gemini, qwen, agent, opencode) — два предложения в стиле When x, do y.
  3. Проверка:

  4. Документация и релиз:

    • глубокий case study в docs/case-studies/issue-1887/ (README, requirements R1–R9, analysis, solutions, existing-components, сохранённые data/);
    • добавлен minor changeset;
    • origin/main влит в ветку; рабочее дерево чистое;
    • PR переименован и переведён в статус ready for review.

This summary was automatically extracted from the AI working session output.

@konard

konard commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $6.227771

📊 Context and tokens usage:

Claude Opus 4.8: (2 sub-sessions)

  1. 116.5K / 1M (12%) input tokens, 35.4K / 128K (28%) output tokens
  2. 63.0K / 1M (6%) input tokens, 12.4K / 128K (10%) output tokens

Total: (20.2K new + 268.1K cache writes + 6.2M cache reads) input tokens, 53.3K output tokens, $6.227771 cost

🤖 Models used:

  • Tool: Anthropic Claude Code
  • Requested: opus
  • Model: Claude Opus 4.8 (claude-opus-4-8)

📎 Log file uploaded as Gist (2842KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard

konard commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

✅ Ready to merge

This pull request is now ready to be merged:

  • All CI checks have passed
  • No merge conflicts
  • No pending changes

Monitored by hive-mind with --auto-restart-until-mergeable flag

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant