Case study: reason for rejection (Codex solve-run on formal-ai #645) + low-reasoning warning

.changeset/issue-2047-codex-low-reasoning-warning.md

@@ -0,0 +1,5 @@
+---
+'@link-assistant/hive-mind': patch
+---
+
+Log a visible `⚠️ Low reasoning` warning when Codex runs with reasoning effort disabled, and add a case study (`docs/case-studies/issue-2047`) investigating a rejected Codex `solve`-run.

docs/case-studies/issue-2047/raw-data/formalai-issue-494.json

@@ -0,0 +1 @@
+{"author":{"id":"MDQ6VXNlcjE0MzE5MDQ=","is_bot":false,"login":"konard","name":"Konstantin Diachenko"},"body":"We need to make sure we keep the longest \n\nOur learned experience (generalized algorithms) and history of recorded raw events, that cannot be extracted from remote sources.\n\nAnd all cached data from external services, all our immediate and intermediate conclusions we do when reason from raw events can be freed if we don't have enough space in AI memory.\n\nWe also need to count usage for cached data, so if something is not used in cached data (even in seed data) it can be freed first.\n\nWe should always free only enough space to store next required links/data/files.\n\nBy default system should free nothing, but when not enough space in memory we should ask user if to enable auto-free-space algorithm as described above.\n\nWhen even our algorithm cannot free enough space for thinking or operation, we should ask user to migrate AI memory to bigger storage.","createdAt":"2026-06-16T09:47:21Z","title":"Free space policy"}

docs/case-studies/issue-2047/raw-data/formalai-issue-540.json

@@ -0,0 +1 @@
+{"author":{"id":"MDQ6VXNlcjE0MzE5MDQ=","is_bot":false,"login":"konard","name":"Konstantin Diachenko"},"body":"Restructure deduplication based on recalculated frequencies of use.\n\nBy default day dreaming is turned on - it works while user uses the app, but in the background, so it never blocks any UI rendering, execution of tasks and so on. It should be executed with lowest possible priority.\n\nAnd also on desktop it should be possible to activate dreaming daemon, that will restructure deduplication based on recalculated frequencies of use to save space, and also find patterns, regularities, trends, laws.\n\nThat process can also do garbage collection, so there always for example at least 20% of free memory / space, so we selectively forget some things, that can be refetched from the external sources, that can be recomputed from raw data of events we have (and we have algorithms for that recomputation).\n\nWe can also use that time for generalizing our algorithms to forget about specific algorithms.\n\nGeneralization should be treated as successful, as given input generalized algorithm can produce specific algorithm, as soon as it is working, specific algorithm can be forgotten.\n\nAs dreaming is like garbage collection + indexing, we should apply fully this task also here https://github.com/link-assistant/formal-ai/issues/494\n\nSo the resulting pull request will close both tasks.\n\nMake sure to be ambitious, take into account our vision, requirements, roadmap (the latest have stronger weight), contributing guide lines, testing guidelines and so on.\n\nIdeally this task itself must be fully and partially solvable by Formal AI connected to Agent CLI. You task to drive Formal AI to make all the actions, if something fails, you improve algorithms by generalization, so Formal AI have enough features to solve each part of this task, and also the task as the whole.\n\nWe need to collect data related about the issue to this repository, make sure we compile that data to ./docs/case-studies/issue-{id} folder, and use it to do deep case study analysis (also make sure to search online for additional facts and data), list of each and all requirements from the issue, and propose possible solutions and solution plans for each requirement (we should also check known existing components/libraries, that solve similar problem or can help in solutions).\n\nPlease plan and execute everything in this single pull request, you have unlimited time and context, as context auto-compacts and you can continue indefinitely, until it is each and every requirement fully addressed, and everything is totally done.\n","comments":[],"createdAt":"2026-06-20T13:23:04Z","title":"Dreaming"}

docs/case-studies/issue-2047/raw-data/formalai-pr-645-review-comments.json

@@ -0,0 +1 @@
+[]

docs/case-studies/issue-2047/raw-data/formalai-pr-645-reviews.json

@@ -0,0 +1 @@
+[]

docs/case-studies/issue-2047/raw-data/formalai-pr-645.json

@@ -0,0 +1 @@
+{"body":"## Summary\n\n- make retained dreaming amendments affect future Chat Completions, Responses, and agentic final answers without requiring the user to repeat a standing requirement\n- replay candidate tasks through the real solver before forgetting covered specifics, mine recurring task structures, and retain stable multilingual amendments and patterns\n- measure the actual memory filesystem, include incoming writes and cached/seed links, request durable cleanup consent, and recommend larger storage when safe reclamation is insufficient\n- run core and desktop dreaming only after 60 seconds of idle time at low process priority, yielding when foreground activity resumes; learning remains enabled while deletion requires explicit consent\n- ground the implementation in a machine-readable recipe and multilingual cues, plus a reproducible Agent CLI gap audit and exact committed session\n\n## How to reproduce the previous gaps\n\n1. Retain a `meta_algorithm_amendment`, then submit a new related task without restating the requirement: previously the amendment was stored but was not read by the answer paths.\n2. Create a covered-specific candidate whose expected answer differs from the old input-plus-rule shortcut: previously it could be marked reproducible without running the real solver.\n3. Trigger memory pressure on a real store: capacity was supplied as static CLI numbers, incoming-write size and cached/seed links were omitted, and desktop work ran on a timer without true idle/yield checks.\n\n## Verification\n\n- `cargo test --all-features` — 1,488 passed, 2 ignored\n- `cargo clippy --all-targets --all-features -- -D warnings`\n- `cargo fmt --check`\n- `git diff --check`\n- `cargo test --all-features memory_maintenance` — 18 passed\n- `cargo test --all-features issue_540_agent_cli` — 3 passed\n- `cargo test --all-features dreaming_meta_algorithm` — 5 passed\n- `npm --prefix desktop test` — 100 passed plus smoke checks\n\n## Review artifacts\n\n- [Agent CLI session](https://github.com/link-assistant/formal-ai/blob/issue-540-daaf4da2188a/docs/case-studies/issue-540/agent-cli-session-dreaming-audit.json)\n- [Generated gap analysis](https://github.com/link-assistant/formal-ai/blob/issue-540-daaf4da2188a/docs/case-studies/issue-540/dreaming-gap-analysis.lino)\n- [Amended acceptance criteria](https://github.com/link-assistant/formal-ai/blob/issue-540-daaf4da2188a/docs/case-studies/issue-540/raw-data/pr-645-amendment-2026-07-10.md)\n- [Library research](https://github.com/link-assistant/formal-ai/blob/issue-540-daaf4da2188a/docs/case-studies/issue-540/raw-data/online-research.md)\n\nCloses #540\nCloses #494\n","commits":[{"authoredDate":"2026-07-08T22:23:15Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-08T22:23:15Z","messageBody":"Adding .gitkeep for PR creation (default mode).\nThis file will be removed when the task is complete.\n\nIssue: https://github.com/link-assistant/formal-ai/issues/540","messageHeadline":"Initial commit with task details","oid":"1ee0a6ff16fc882edfd4e46418aae913c8c7fcd9"},{"authoredDate":"2026-07-08T23:03:35Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-08T23:03:35Z","messageBody":"","messageHeadline":"Add dreaming memory maintenance planner","oid":"7a8096cd4e1559f749d282cce588a7afc655ad21"},{"authoredDate":"2026-07-08T23:11:25Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-08T23:11:25Z","messageBody":"","messageHeadline":"Fix dreaming clippy warnings","oid":"1cba7ede1de811bd07c253a7eeaf53295c9cecb2"},{"authoredDate":"2026-07-08T23:45:42Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-08T23:45:42Z","messageBody":"","messageHeadline":"Keep the formal-ai CLI entrypoint under the line limit","oid":"48b1d212150ff9d0f3451b3e5f81f09af46f5a55"},{"authoredDate":"2026-07-09T00:18:36Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-09T00:18:36Z","messageBody":"This reverts commit 1ee0a6ff16fc882edfd4e46418aae913c8c7fcd9.","messageHeadline":"Revert \"Initial commit with task details\"","oid":"cc1d375a432f4490c1c24cc46e8ab479906cead5"},{"authoredDate":"2026-07-09T09:25:36Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-09T09:25:36Z","messageBody":"While idle, dreaming now recalculates the topics the user interacts\nwith most, remembers the durable requirements stated on them, and\ngeneralizes each into a retained meta_algorithm_amendment so the user\nnever repeats himself. Specifics a retained amendment can reproduce are\nforgotten first under pressure (ForgetCoveredSpecific), while the\ngeneralization is kept forever. The dreaming meta-algorithm is recorded\nas grounded data in data/meta/dreaming-recipe.lino, pinned to the live\nsource by tests/unit/specification/dreaming_meta_algorithm.rs.","messageHeadline":"Teach dreaming to learn topics and generalize requirements","oid":"4b4b92a486a2911b0d90caa41918521d53cf89a2"},{"authoredDate":"2026-07-09T09:25:47Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-09T09:25:47Z","messageBody":"","messageHeadline":"Merge remote-tracking branch 'origin/main' into issue-540-daaf4da2188a","oid":"ebbcbe74e27f4fa4b91357b43f9f634f7592f779"},{"authoredDate":"2026-07-10T19:03:44Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-10T19:03:44Z","messageBody":"","messageHeadline":"Implement verified dreaming and storage runtime","oid":"45e3dfbc56fb8554578e5d5dbc669396b1d4b49d"},{"authoredDate":"2026-07-10T19:03:50Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-10T19:03:50Z","messageBody":"","messageHeadline":"Add idle-aware desktop dreaming consent","oid":"7027fcfd75370285ce0a86bd2cdbac60344709b3"},{"authoredDate":"2026-07-10T19:04:07Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-10T19:04:07Z","messageBody":"","messageHeadline":"Ground dreaming audit and refresh generated artifacts","oid":"d153d4f0992bd39aa31bb4b4182974873a794195"},{"authoredDate":"2026-07-10T19:04:12Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-10T19:04:12Z","messageBody":"","messageHeadline":"Merge remote-tracking branch 'origin/main' into issue-540-daaf4da2188a","oid":"156615bb23963e77426dc97a1c4b76306dad3014"},{"authoredDate":"2026-07-10T19:09:23Z","authors":[{"email":"drakonard@gmail.com","id":"MDQ6VXNlcjE0MzE5MDQ=","login":"konard","name":"konard"}],"committedDate":"2026-07-10T19:09:23Z","messageBody":"","messageHeadline":"Fix dreaming metadata whitespace","oid":"46b57fd80bc350d06dc474194bd317dd3e3ce525"}],"createdAt":"2026-07-08T22:23:23Z","headRefName":"issue-540-daaf4da2188a","state":"OPEN","title":"Complete verified idle dreaming and memory cleanup"}

docs/case-studies/issue-2047/raw-data/issue-2047.json

@@ -0,0 +1 @@
+{"author":{"id":"MDQ6VXNlcjE0MzE5MDQ=","is_bot":false,"login":"konard","name":"Konstantin Diachenko"},"body":"https://gist.githubusercontent.com/konard/265951976ffb71c3e2396b417e523cf0/raw/cd507838a1f611ef8a408a20930b8148a88de25e/tmp-solution-draft-log-pr-1783784936003.txt.log.txt\n\nWe need to download all logs and data related about the issue to this repository, make sure we compile that data to `./docs/case-studies/issue-{id}` folder, and use it to do deep case study analysis (also make sure to search online for additional facts and data), in which we will reconstruct timeline/sequence of events, list of each and all requirements from the issue, find root causes of the each problem, and propose possible solutions and solution plans for each requirement (we should also check known existing components/libraries, that solve similar problem or can help in solutions).\n\nIf there is not enough data to find actual root cause, add debug output and verbose mode if not present, that will allow us to find root cause on next iteration.\n\nIf issue related to any other repository/project, where we can report issues on GitHub, please do so. Each issue must contain reproducible examples, workarounds and suggestions for fix the issue in code. Also double check to fully apply requirements to entire codebase, so if we have issue in multiple places, it should be fixed in all them.\n\nPlease plan and execute everything in this single pull request, you have unlimited time and context, as context auto-compacts and you can continue indefinitely, until it is each and every requirement fully addressed, and everything is totally done.","createdAt":"2026-07-11T21:33:13Z","title":"Investigate the reason for rejection"}