add retrocheck.py

[robertnurnberg]

The new script allows to systematically and automatically evaluate the "retrograde analysis" capabilities of an engine. It works similarly to the matecheck.py script from https://github.com/vondele/matetrack, but before the final search for each position it fills the engine's hash table by performing searches long the given mating line, going backwards from the checkmate to the puzzle position.

A clear improvement can be seen comparing "hash filling" with 1 node searches compared to 1000 node searches.

> python retrocheck.py --nodesFill 1 --nodes 1000
Loaded 6554 FENs, with max(|bm|) = 126.

retrocheck started for ./stockfish on matetrackpv.epd with --nodes 1000 --nodesFill 1 ...
100%|###########################################| 33/33 [02:31<00:00,  4.59s/it]

Using ./stockfish on matetrackpv.epd with --nodes 1000 --nodesFill 1
Engine ID:     Stockfish dev-20260106-c27c1747
Total FENs:    6554
Found mates:   137
Best mates:    119

> python retrocheck.py --nodesFill 1000 --nodes 1000
Loaded 6554 FENs, with max(|bm|) = 126.

retrocheck started for ./stockfish on matetrackpv.epd with --nodes 1000 --nodesFill 1000 ...
100%|###########################################| 33/33 [04:19<00:00,  7.86s/it]


Using ./stockfish on matetrackpv.epd with --nodes 1000 --nodesFill 1000
Engine ID:     Stockfish dev-20260106-c27c1747
Total FENs:    6554
Found mates:   1553
Best mates:    1446

Setting any of the fill limits to zero is equivalent to running matecheck.py:

> python retrocheck.py --nodesFill 0 --nodes 1000
Loaded 6554 FENs, with max(|bm|) = 126.

retrocheck started for ./stockfish on matetrackpv.epd with --nodes 1000 --nodesFill 0 ...
100%|###########################################| 33/33 [01:04<00:00,  1.95s/it]


Using ./stockfish on matetrackpv.epd with --nodes 1000 --nodesFill 0
Engine ID:     Stockfish dev-20260106-c27c1747
Total FENs:    6554
Found mates:   61
Best mates:    41

> python ../matetrack/matecheck.py --nodes 1000
Loaded 6554 FENs, with max(|bm|) = 126.

Matetrack started for ./stockfish on matetrack.epd with --nodes 1000 ...
100%|###########################################| 33/33 [00:44<00:00,  1.35s/it]


Using ./stockfish on matetrack.epd with --nodes 1000
Engine ID:     Stockfish dev-20260106-c27c1747
Total FENs:    6554
Found mates:   61
Best mates:    41

Edit: Numbers fixed after the application of 2221195.

[pb00068]

Hi Robert,

to have a better and more valuable measurment of the "retrograde analysis" capability I suggest to count not only the occurences when the mate is still found on the starting (=puzzle) position, but also count each time SF finds (regains) the mate again along the mating line while going backwards.
Also it is useless work IMO to stick on going backwards until the starting position once SF has lost the path to mate while stepping back.

[robertnurnberg]

Thanks for your feedback. I can think about adding this as an option.

Note that for now the PV along which the hash is filled need not be a complete mating line at all. In theory it can be any sequence of moves.

[pb00068]

Yes, but my point is, that once you get lost about the mating line while stepping back, it's very improbable that when stepping back you suddenly again regain the mating line.
Lets say we have SF found -M22 by searching 1m nodes,
Then you step back 1 halfmove, search again 1m nodes and get +800 cp as final value.
Now it's very improbably IMO that when stepping back another move and search 1m nodes again you now get a -Mx eval again.
It might happen by accident but it is then not very meaningful.
It's more probably that all subsequent searches until the starting position don't find the mate anymore,
also this (= counting each try as failure) is then not very meaningful (by the way you then also you waste computing time).
IMO it's better to abort immediately to have a meaningful quantitative(!) measurment of retrograde analysis capability.

[robertnurnberg]

The idea of the script is that you somehow pre-fill the hash table with shallow searches in the subtree, before you do the actual analysis of the puzzle. In particular, usually you would use a higher nodes/depth limit at root than along the given PV.

What you have in mind is implemented in the script provepvs.py. For example, there the search depth is linearly increased from leaf towards the root, and search can be performed until the mate is found.

But I wanted to keep the retrocheck.py script as simple as possible for now.

[pb00068]

@robertnurnberg There must be something wrong with your current version of retrocheck.py

It reports failings on simpliest problems:
I verified problem with FEN 8/B3p3/2K1p1p1/4Pkpb/5pn1/2P2N1B/4PP2/6R1 w - - bm c6d7; PV: c6d7 f5e4 d7e6 g4h2 f3g5;
Both engines master and patch are able to find the M3 well below 1.000.000 nodes directly from the starting pos.
If I simulate the whole procedure with Chess Arena it easily finds the mate:

1. Open Arena GUI and analyze FEN 8/B3p3/2K1p1p1/4Pkpb/5pn1/2P2N1B/4PP2/6R1 w - -
After mating PV is found append the variation to the end (can be choosen after right mouse click on engine window)
2. Go to one move before the mating position
3. Under Levels -> Adjust -> choose Nodes set to 100.000 (=nodes fill rate)
4.Restart the engine and press CTRL-Z, this will submit to the engine
go nodes 100000
5.Move one move back and press CTRL-Z again
6.Repeat it until the starting pos is reached
7.Under Levels -> Adjust -> choose Nodes set to 1.000.000 (=nodes rate)
8. press CTRL-Z again and you will see the mate in 3 will be found.

In contrast the retrocheck.py script reports this problem as fail:

py retrocheck.py --engine ..\sf\stockfish.exe --hash 128 --nodes 1000000 --nodesFill 100000
Loaded 1 FENs, with max(|bm|) = 3.

retrocheck started for ..\sf\stockfish.exe on matetrackpv.epd with --nodes 1000000 --nodesFill 100000 --hash 128 ...

Using ..\sf\stockfish.exe on matetrackpv.epd with --nodes 1000000 --nodesFill 100000 --hash 128
Engine ID:     Stockfish dev-20260113-nogit
Total FENs:    1
Found mates:   0
Best mates:    0

PS.: After adding

with engine.analysis(board, self.limitFill, game=board) as analysist:
                        for info in analysist:
                            dummy= ""

the script seems work correcty:

Using ..\sf\stockfish.exe on matetrackpv.epd with --nodes 1000000 --nodesFill 100000 --hash 128
Engine ID:     Stockfish dev-20260113-nogit
Total FENs:    1
Found mates:   1
Best mates:    1

[robertnurnberg]

Thanks! Should be fixed with 2221195.

[pb00068]

OK, thanks.
Can you correct the comments/results on
official-stockfish/Stockfish#6531 (comment)
with the fixed version?