Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 76 additions & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,21 @@
name: tests

# The gate, run for real. tests/run.sh drives the actual config in headless
# nvim and on real ptys, so this is the same 556 cases you get locally -- not a
# nvim and on real ptys, so this is the same suite you get locally -- not a
# lint. It exists because "works on Linux" was a claim in the README that
# nothing checked, and on a stock Ubuntu box it was not true.
#
# Two platforms, because both are claimed and neither can stand in for the
# other. They are not the same run with a different logo:
#
# linux the OS branches added by the WSL port -- the clipboard fallbacks,
# the sound and notification chains, the /mnt python3 rejection
# macos the branches those were added AROUND, which until now were only
# ever "unchanged by construction". It is also the only job that
# reaches bash 3.2, the BSD pgrep/ps/sed/awk, and the perl alarm leg
# of tests/bin/timeout -- on Linux the shim always finds a real
# timeout(1) and hands over, so that leg has never run in CI at all.
#
# Every PR, and main. NOT every push: a branch with a PR open would then run the
# whole 6-minute gate twice for one commit, once per event. A feature branch
# pushed with no PR open is the deliberate gap -- open the PR and it is covered.
Expand Down Expand Up @@ -94,3 +105,67 @@ jobs:
out=$(SPEED=0.4 bash -c "$real"'; secs 0.09')
[ "$out" = "0.0360" ] || { echo "demo.sh secs() returned '$out', expected 0.0360"; exit 1; }
echo "demo.sh secs() -> $out"

macos:
# arm64. Not a matrix with an Intel leg: the Intel runner labels are on
# GitHub's deprecation path, so it would be a check that needs re-pinning
# rather than one that keeps working, and nothing here is arch-sensitive.
runs-on: macos-latest
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false

# The same v0.11.0 the linux job pins, from the same release, checked
# against the same shasum.txt. brew would have been one line, but then
# "which nvim did the gate run against" has a different answer every week
# and a macOS-only failure could be the config or could be the editor.
- name: neovim
run: |
set -euo pipefail
curl -fsSL -o nvim.tar.gz \
https://github.com/neovim/neovim/releases/download/v0.11.0/nvim-macos-arm64.tar.gz
echo "20055d322e80194350ee0e47f1ba5a8016d1cdbe7e283d62f882f640326e18a7 nvim.tar.gz" | shasum -a 256 -c -
# curl does not set com.apple.quarantine, so this is belt and braces
# -- but it is what neovim's own install notes say to do, and a
# Gatekeeper refusal would surface as "nvim: bad interpreter".
xattr -c nvim.tar.gz || true
sudo tar xzf nvim.tar.gz -C /opt
sudo ln -sf /opt/nvim-macos-arm64/bin/nvim /usr/local/bin/nvim
nvim --version | head -1

# fzf only. Everything else the gate needs is already the operating
# system, and each omission is deliberate:
#
# zsh, perl, python3, pgrep in the base image (python3 is ASSERTED
# below rather than installed, exactly as
# the linux job does it)
# clipboard pbcopy/pbpaste ARE macOS. Whether they
# work in a runner's session is a separate
# question, and tests/run.sh now probes it
# xvfb Quartz needs no display server from us
# coreutils would install gtimeout, not timeout, so
# tests/bin/timeout would not find it
# anyway -- and taking the perl leg is the
# entire point of running here
- name: deps
run: brew install fzf

- name: the gate
run: |
command -v python3 >/dev/null || { echo "no python3 on the runner"; exit 1; }
./tests/run.sh

# Same check as the linux job, and this is the run of it that matters:
# secs() exists to drop bc, and sed/awk/bash here are the BSD and 3.2
# ones. An awk expression that only GNU awk accepts passes on ubuntu and
# fails on the machine the demo is actually recorded on.
- name: demo.sh parses, and needs no bc
run: |
set -euo pipefail
bash -n demo.sh
real=$(sed -n '/^secs() {/p' demo.sh)
[ -n "$real" ] || { echo "could not find secs() in demo.sh"; exit 1; }
out=$(SPEED=0.4 bash -c "$real"'; secs 0.09')
[ "$out" = "0.0360" ] || { echo "demo.sh secs() returned '$out', expected 0.0360"; exit 1; }
echo "demo.sh secs() -> $out"
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ work, because there is no unsaved state to lose.
## Nothing helps you type

No autocomplete. No LSP, no Copilot, no snippets, no signature hints, no
auto-import. Not "turned off" — **absent, at every source, and a 556-case test
auto-import. Not "turned off" — **absent, at every source, and a 558-case test
suite asserts it**. Syntax highlighting is the only thing on screen besides your
own keystrokes. Your `~/.config/nvim` and `~/.vimrc` are never touched; the
config loads with `nvim -u` and exists only inside drill.
Expand Down Expand Up @@ -149,7 +149,7 @@ so `git pull` stays clean forever.
| [KEYS.md](KEYS.md) | every keybinding, every conflict, and how each is resolved |
| [docs/design.md](docs/design.md) | why the editor behaves the way it does — the mouse, autosave, search, quitting, and the five deliberate trade-offs |
| [docs/verify.md](docs/verify.md) | prove nothing is helping you, and that your own nvim is untouched |
| [docs/testing.md](docs/testing.md) | the 556-case gate that drives the real config with real keycodes |
| [docs/testing.md](docs/testing.md) | the 558-case gate that drives the real config with real keycodes |
| [docs/recording.md](docs/recording.md) | `demo.sh` — the self-driving tour that recorded the video above |
| [docs/wsl.md](docs/wsl.md) | WSL: the clipboard, the timer's sound and notification, and the one thing you have to check by hand |

Expand Down
2 changes: 1 addition & 1 deletion TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ Landed as a stack of PRs, one per checkpoint, each based on the previous.
What I had actually measured was a screenless nvim, where `confirm()`
cannot draw and returns instantly. The skip built on that reasoning is
reverted; the check runs unconditionally, as it always did.
*Gate: full `./tests/run.sh` 556/556; `./demo.sh --check` clean.*
*Gate: full `./tests/run.sh` 558/558; `./demo.sh --check` clean.*
- [x] **5 — `wsl/05-ci-docs`** · `.github/workflows/tests.yml` running the suite
on `ubuntu-latest` under `xvfb` with `xclip`, so the clipboard cases are
real; `docs/wsl.md`; requirement lines in `README.md`, `KEYS.md`,
Expand Down
48 changes: 35 additions & 13 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ and diffing what comes out. Nothing is mocked: if a case passes, that keystroke
does that thing in this config.

```sh
./tests/run.sh # 556 cases; exit 0 only if all pass
./tests/run.sh # 558 cases; exit 0 only if all pass
./tests/run.sh sel_ # just the select-mode cases
```

| Suite | Cases | What it drives |
|---|---|---|
| `suite_options.sh` | 84 | config invariants no key-driven test can see: completion off at every source, zero LSP clients, no swap/backup/undo files, the cursor shape in both panes, and every mapping registered in the modes it claims — including that `Ctrl+/` is *not* bound in terminal mode and `Ctrl+C` is *not* bound in normal or terminal, so SIGINT still reaches a running program |
| `suite_options.sh` | 86 | config invariants no key-driven test can see: completion off at every source, zero LSP clients, no swap/backup/undo files, the cursor shape in both panes, and every mapping registered in the modes it claims — including that `Ctrl+/` is *not* bound in terminal mode and `Ctrl+C` is *not* bound in normal or terminal, so SIGINT still reaches a running program |
| `suite_config.sh` | 30 | headless nvim: shift+arrow selection, `Tab`/`Shift+Tab`, cut/paste/select-all, undo/redo, `Ctrl+C` copying without losing the selection |
| `suite_mouse.sh` | 58 | pty: click to caret from every mode, into empty space and past EOF, jitter in both axes, real drags, double-click, Option/Ctrl+click, and clicks between the file and the interpreter |
| `suite_search.sh` | 36 | `Ctrl+F`, exactly when the highlight and hints appear and go, and the Esc chain — including that it resumes typing at the exact column. Headless **and** pty: the Esc chain cannot be seen headlessly, because feedkeys force-ends Insert as the typeahead drains |
Expand Down Expand Up @@ -67,8 +67,18 @@ macOS. `tests/bin/timeout` is that `alarm` shim; `run.sh` puts it on `PATH`
first, so on Linux it hands over to the real `timeout` rather than shadowing it
with something weaker.

CI runs the same script on `ubuntu-latest` under `xvfb` with `xclip`, so the
clipboard cases are real there too — see `.github/workflows/tests.yml`.
CI runs the same script on **both platforms this project claims** — see
`.github/workflows/tests.yml`. `ubuntu-latest` under `xvfb` with `xclip`, and
`macos-latest` with the pasteboard it already has, so the clipboard cases are
real in both. Both jobs pin the same nvim 0.11.0 by tarball and checksum, so a
job that goes red is the code and not the editor.

They are not the same run twice. The Linux job covers the branches the WSL port
added; the macOS job covers the ones they were added around, and is the only
place `bash` is 3.2, `pgrep`/`ps`/`sed`/`awk` are the BSD ones, and
`tests/bin/timeout` actually takes its `perl` leg — on Linux the shim always
finds a real `timeout(1)` and hands over, so that path had never run in CI until
the macOS job existed.

## One known flake, on WSL only

Expand All @@ -78,10 +88,16 @@ Roughly one full-suite run in three **on WSL**, one of the six cases in
and there is no reason the other three are immune. It is always the same
signature: **the buffer is correct and `+` reads back empty**.

They pass when run alone (10/10, 3/3, 3/3 for the three above), and the
instrumented mapping shows `getregion` returning the right text every time — so
the selection and the keybinding are fine. What intermittently fails is the
write to the *system* clipboard under the load of a 556-case run.
The instrumented mapping shows `getregion` returning the right text every time,
so the selection and the keybinding are fine. What intermittently fails is the
write to the *system* clipboard.

**Load makes it worse but is not the cause.** This page used to say these cases
Comment thread
jainal09 marked this conversation as resolved.
Outdated
pass when run alone, on 10/10 and 3/3 samples. Bigger samples say otherwise:
`ctrlc_visual_charwise_exclusive`, run by itself in a loop, failed **2 times in
30** with the same signature. So isolation lowers the rate by roughly a factor
of five — it does not reach zero, and a single solo pass proves less than it
looks like it does.

It is not the provider. It reproduces with `wl-copy` (WSLg's default pick) and
again with `WAYLAND_DISPLAY` unset to force `xclip`, and in the same run one
Expand All @@ -91,16 +107,22 @@ single `xclip` under `xvfb` is evidently steadier than a WSLg session.

It is deliberately not papered over. A retry would hide a real clipboard
regression, and giving these cases a stub provider would break the one rule the
suite has: nothing here mocks anything. If you see this signature — buffer right, `+` empty — re-run that single case
before believing it. For example:
suite has: nothing here mocks anything. If you see this signature — buffer
right, `+` empty — re-run that single case **several times** before believing
it. Once is not enough, for the reason above:

```sh
./tests/run_test.sh --name check --content 'aa\nbb\ncc' \
--keys '<S-Down><S-Down><C-c>Z' --expect 'Zcc' --expect-reg '+=aa
for i in 1 2 3 4 5; do
./tests/run_test.sh --name "check$i" --content 'aa\nbb\ncc' \
--keys '<S-Down><S-Down><C-c>Z' --expect 'Zcc' --expect-reg '+=aa
bb'
done
```

If it passes alone, you saw the flake. If it fails alone, you have a real bug.
Read it as a rate, not a verdict. **Five passes** and you saw the flake.
**Five failures** and you have a real bug. One failure in five is the flake
again — at 2-in-30 you should expect to meet it eventually, and treating that
as a regression will send you hunting for a bug that is not there.

## Diagnostics

Expand Down
2 changes: 1 addition & 1 deletion docs/wsl.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ Ctrl+click are both just clicks, not vim surprises.
## Running the tests

```sh
./tests/run.sh # 556 cases
./tests/run.sh # 558 cases
```

Needs a clipboard provider on PATH; `run.sh` warns up front if there is none,
Expand Down
86 changes: 74 additions & 12 deletions tests/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,25 @@
#
# Requires: nvim 0.9+, zsh (the timer suite runs drill.sh under bash AND
# zsh -- they disagree about word splitting), python3 (the mouse suite needs a
# pty; headless nvim has no screen grid to click on), and a clipboard provider
# -- wl-copy/xclip/xsel/win32yank, or clip.exe on WSL -- because several cases
# assert the real '+' register. perl is needed only where there is no
# pty; headless nvim has no screen grid to click on), and a WORKING clipboard
# provider -- pbcopy on macOS, wl-copy/xclip/xsel/win32yank elsewhere, or
# clip.exe on WSL -- because several cases assert the real '+' register. It is
# probed, not assumed: pbcopy over ssh and xclip with no display are both
# installed and both useless. perl is needed only where there is no
# timeout(1) at all, i.e. macOS; everywhere else bin/timeout hands over to the
# real one. See NOTES.md for why the harness is shaped the way it is.
# ============================================================================
set -uo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
FILTER="${1:-}"

# Same PATH prefix every suite sets for itself, hoisted here because the
# clipboard preflight below bounds its probes with `timeout` and macOS has no
# timeout(1) -- without this the probe would fail as "command not found" on the
# one platform whose branch was added to be probed. On Linux bin/timeout finds
# the real one and hands over, so nothing about this run changes there.
PATH="$DIR/bin:$PATH"; export PATH

export DRILL_CONFIG="${DRILL_CONFIG:-$(cd "$DIR/.." && pwd)/nvimrc.lua}"

if [ ! -f "$DRILL_CONFIG" ]; then
Expand All @@ -34,9 +43,55 @@ command -v nvim >/dev/null || { echo "run.sh: nvim not on PATH" >&2; exit 2; }
# The config sets clipboard=unnamedplus, and several cases in suite_config.sh
# assert the REAL '+' register. With no provider those fail for a reason that
# has nothing to do with the keybinding under test, and read as a regression in
# the mapping. Say it once, up front, instead. macOS always has pbcopy, so this
# is silent there.
if [ "$(uname -s)" != "Darwin" ]; then
# the mapping. Say it once, up front, instead.
CLIP=""
if [ "$(uname -s)" = "Darwin" ]; then
# pbcopy IS macOS -- it cannot be missing, which is why this used to skip the
# whole preflight and say so. But present is not working: the pasteboard lives
# in a per-session pbs, so over ssh or on a CI runner pbcopy can be right there
# on PATH and still have nothing to talk to. That is the same mistake the
# xclip branch below already stopped making ("a display name is not a
# display"), so make it once, in one shape, on both platforms.
#
# Both halves, because the register cases need copy AND paste, and pbpaste is
# the one that actually reads back from pbs.
#
# -pboard find, NOT the general pasteboard. macOS has four named pasteboards
# served by the same pbs, so `find` proves reachability exactly as well while
# leaving whatever you copied alone -- the general one is destroyed later in
# the run by the register cases themselves, but only if you actually run
# them, and `./tests/run.sh timer` should not cost you your clipboard. The
# non-destructive spelling exists here and has no equivalent below: xsel and
# wl-copy have no unused selection to borrow (wl-copy offers only clipboard
# and primary, and primary is the user's too), so that branch still writes,
# as its own comment already owns.
#
# Saving and restoring the GENERAL pasteboard instead would be worse than the
# bug: pbpaste yields text, so a round-trip silently flattens an image, RTF or
# a file promise to nothing. On the find board that objection does not apply --
# it holds the system find string, which is text by definition -- so this one
# IS put back, and the probe costs you nothing at all. (Worst case a trailing
# newline: $(...) strips them. That is a search term, not your data.)
#
# Compared by VALUE, not by exit code. What the register cases need is a real
# round trip -- write it, read the same thing back -- and that is the property
# to assert, not that two processes happened to exit 0.
#
# The save GATES the write, and that is the whole safety argument: never put
# something on a board you have not proved you can put back. Reading first is
# free, because a failed read is already the answer -- if pbs will not answer
# a paste, it is not a working provider, CLIP stays empty, and we have written
# nothing. Restoring unconditionally instead had a real hole: pbpaste hitting
# the 5s timeout on a merely SLOW pbs leaves FIND_WAS empty, and the restore
# then clears a find term that was there all along.
if FIND_WAS="$(timeout 5 pbpaste -pboard find 2>/dev/null)"; then
if printf x | timeout 5 pbcopy -pboard find >/dev/null 2>&1 &&
[ "$(timeout 5 pbpaste -pboard find 2>/dev/null)" = x ]; then
CLIP="pbcopy"
fi
printf %s "$FIND_WAS" | timeout 5 pbcopy -pboard find >/dev/null 2>&1
fi
else
# Mirror what nvimrc.lua accepts, or this warns when the editor is fine and
# stays quiet when it is not. Two things that means: the same provider list,
# and the same display requirement -- an installed xclip with no DISPLAY
Expand All @@ -46,10 +101,9 @@ if [ "$(uname -s)" != "Darwin" ]; then
# and then fails with "Can't open display", producing exactly the register
# failures this warning exists to explain while the warning stays silent.
# nvimrc.lua cannot afford this probe -- it would be a subprocess on every
# editor start -- but run.sh pays it once per suite, before 556 cases.
# editor start -- but run.sh pays it once per suite, before the whole gate.
# It writes, because a read cannot tell "no display" from "empty clipboard";
# the suite clobbers the clipboard wholesale anyway.
CLIP=""
for c in wl-copy xclip xsel win32yank.exe lemonade doitclient; do
command -v "$c" >/dev/null 2>&1 || continue
case "$c" in
Expand Down Expand Up @@ -77,12 +131,20 @@ if [ "$(uname -s)" != "Darwin" ]; then
timeout 15 powershell.exe -NoProfile -NoLogo -Command Get-Clipboard >/dev/null 2>&1; then
CLIP="clip.exe"
fi
if [ -z "$CLIP" ]; then
echo "run.sh: WARNING -- no clipboard provider on PATH."
echo " The cases asserting register '+' will fail for that alone."
fi

if [ -z "$CLIP" ]; then
echo "run.sh: WARNING -- no working clipboard provider."
echo " The cases asserting register '+' will fail for that alone."
if [ "$(uname -s)" = "Darwin" ]; then
# Nothing to install. A Mac in this state has no pasteboard server to
# reach, which is a property of the session, not of the machine.
echo " pbcopy/pbpaste are installed but did not answer -- no user"
echo " session (ssh, or a CI runner) is the usual reason."
else
echo " apt install wl-clipboard xclip (see docs/wsl.md on WSL)"
echo
fi
echo
fi

echo "config: $DRILL_CONFIG"
Expand Down
Loading
Loading