Romero’s FinalGen of 2012 creates designer endgame tables for specific chess positions that feature no more than one non-pawn piece per side. Larger hard discs and faster solid-state discs have extended the reach of this software and encouraged its greater use. Some new discoveries illustrate here what is now feasible and how FinalGen may be combined with other tools to reach definitive and likely truths.
Pedro Pérez Romero (2012) published the software FinalGen to create position-specific ‘EGT’ endgame tables for chess positions meeting one criterion. Neither side must have more than one piece other than the king: the remaining men must be pawns. The software uses the fact that pawns’ future positions are highly restricted. It is greatly helped if pawns are advanced, facing each other and in the extreme, blocking each other.
Since 2012, computer hardware and software have improved. ‘HDD’ hard discs have increased capacity and offer better GiB/$. ‘SSD’ solid-state discs are replacing HDDs for operational purposes: they too are providing greater capacity and their life expectancy is improving. SSDs also fail more gracefully than HDDs that can crash in dramatic fashion. USB3.2 has superseded USB2.0 and 16-lane PCIe 5.0 will supersede today’s commonly available 4-lane PCIe 3.0. These advances greatly facilitate and encourage the greater use of FinalGen, increasing the number and complexity of positions that it may feasibly address on a specific computer.
Section 1 defines the core concepts and required notation. Sections 2–7 investigate results on specific positions addressed by the authors, including Romero’s own FinalGen tutorial examples. Our full chess analyses cannot be accommodated here but are available within a pgn file and, more fully illustrated and annotated, in the supplementary document with the repository version of this note (Müller and Haworth, 2019), along with the full statistics of the FinalGen computations.
Key concepts and notation
Notation is needed to refer to various position depth metrics, expressible in moves or plies. Strategies for choosing moves should defend the theoretical value of the position and this is assumed. Here it should be noted that FinalGen’s ‘win or draw’ verdict introduces a notional value of ¾ points! In a decisive position, and sometimes in a drawn position, move-optimality may further be determined by a strategy successively minimising (or maximising) depth using a set of depth metrics. Moves may be the only legal move or the only value-preserving move. In the context of a defined strategy, moves may be uniquely optimal, equi-optimal or sub-optimal. The amount by which a move is optimal (sub-optimal) indicates the positional-strength preserved (lost). It should be noted that depth-optimisation is not the only fruit. In chess studies and in games, the defender should be making the move that makes it hardest for the attacker to find the winning move. This is not necessarily the depth-maximising move which, in any case, can vary with metric and sometimes makes no chessic sense at all. Further, a winning strategy will be easier to understand if it does not take advantage of every helpful fortuity.
Depth metric notation:
DTF: ‘Depth according to FinalGen’, usually to a conversion before a clear win,
DTM, DTC: ‘Depth to Mate’, ‘Depth to Conversion’ (i.e., to capture, conversion and/or mate),
DTZ: ‘Depth to Zeroing (of the ply count)’, i.e., to pawn-push, capture and/or mate,
DTZ50′: ‘Depth to Zeroing of ply count in the context of the FIDE 50-move draw-claim rule,
dtc/f/m/z/z50′ 9p(m): an actual position DTC/F/M/Z/Z50′-depth of 9 plies (moves).
‘Preserving value’ is done in the context of FinalGen’s ‘win or draw’ and of the five (rather than three) values of the ‘syzygy’ EGTs. De Man added ‘frustrated win’ and ‘saved loss’ values in between ‘win’, ‘draw’ and ‘loss’.
and minimising dtf, dtm and dtz in that order,
: a strategy preserving value and maximising both dtf and dtm,
: defined strategies for White and for Black,
, and similarly, .
Move optimality, sub-optimality and evaluation:
a value-preserving, equi-optimal move (not noted in our pgn files),
a value-preserving, uniquely-optimal move (‘!’ in our pgn files),
a uniquely value-preserving move (‘!!’ in our pgn files),
move conceding 4 ply in terms of the DTF metric, ‘+(-)’ for attacker (defender),
the only legal move,
a move ‘dropping ¼ point’, e.g., FinalGen finesses ‘win’ ↔ ‘win or draw’,
a move dropping ½ point, i.e., ‘win’ ↔ ‘draw’ or ‘draw’ ↔ ‘loss’, and
Eds+0.97 ≡ an engine (here, Fritz14) evaluation of +0.97 at nominal search-depth of s plies.
Further, it is worth noting that our use of ‘!’ and ‘!!’ in the pgn files is unusual, and in particular, that these symbols are not emoticons marking notably perceptive and/or surprising moves as is the custom. Indeed, many of the moves denoted in this way are entirely obvious.
Positions 01-10b: Romero’s original FinalGen demonstrations
Romero (2012) provides eleven endgame positions. Four of these – R01/06/10a/10b – are admittedly directly evaluable using online ‘EGT’ endgame tables (de Man et al., 2018; Guo, 2018; Lomonosov, 2012). However, these examples are a convenient introduction to FinalGen and show where SF/M/Z strategies diverge. A further five – R03/05/07/08/09 – feature eight men and are arguably in range for albeit fallible, deep-searching, multi-threaded chess engines with 7- or 6-man EGTs.
Finalgen creates EGTs in two modes, ‘Normal’ (‘N’ below) and ‘Search for Draw’ (‘SfD’). ‘SfD’ is more efficient in both space and time than ‘N’ and only returns a ‘Draw’ verdict if this is the case.
The second author reproduced all the Romero (2012) results apart from those for R02 and R10b where the currently downloadable FinalGen v1.4 gives an indecisive ‘win or draw’ evaluation. Rusz (2018) confirms that a previous version created the results on the website. The supplementary document investigates why FinalGen does not see the win in these two examples.
PC B: Razer Blade 15 (2018), x64, 2.2GHz Core i7-8750H/16GiB, 0.5TiB PM961 SSD via M.2 PCIe-3 x4.
in both ‘N’ and ‘SfD’ modes and with EGT files NTFS-compressed or not. The computational experiments were not run under strict benchmark conditions, e.g., with defined clock-rate and controlled temperatures. The space figures are accurate to five figures4
Windows7/HDD and Windows10/SSD agree on file ‘size’ but vary in ‘size on disc’ by about 1 in 1,000,000.
but the time figures are reasonably indicative rather than exactly reproducible. Space and time constraints did not allow all eight options to be pursued for every position of interest.
Fig. 2 includes Romero’s 2012 evaluations of the positions with Finalgen v1. The appendix notes some detailed logistics but the headlines are:
Romero’s examples, all wtm: (a) R02 1-0, (b) R03 =, (c) R04 1-0, (d) R07 = and (e) R08 1-0.
SfD mode elapsed times are no more than 43% of ‘N’ elapsed times,
SfD-EGTs are no more than 8% of the size of N-EGTs,
SfD mode indicates ‘won or drawn’ on all won and some drawn positions,
SfD is therefore a good ‘first try’ at definitively evaluating a drawn endgame,
elapsed times with SSDs can be as little as 6% of previous times with HDDs,
N-EGTs compress reliably to ∼29%, and SfD-EGTs to ∼39% of their intrinsic size,
On PC B, FinalGen took ∼40% longer with compressed N-EGTs, and
On PC B, FinalGen took ∼7% longer with compressed SfD-EGTs.
Positions analysed here with FinalGen on PC B.
Positions 11a-11b: Navrotescu–Shah, Pays de Charleroi Open, 2017, round 4
This game featured in the Chessbase (2018) Endgame Magic 110 as the finale of a rich and fascinating discussion between the first author and IM Sagar Shah. Navrotescu lost in Fig. 3a with 56. Rd3?? as 56. Bc4!! draws. Even so, the win was not simple and White need not have resigned on move 65: the endgame is a big fish which still has to be landed. The position after 64. . . . Rxf4 attracted the sharp minds and serious attention of GMs Tigran Gharamian, Sandipan Chanda and Jacob Aagaard. For months, the win was not at all clear until Jacob attacked it with FinalGen. It was then clear why the win was not clear: it is exceptionally deep. Sandipan then commented on the themes that emerged, noting that Black must prevent White from establishing its king on f4 and its rook on a4. Perhaps, looking past the sometimes-distracting detail of the computer’s lines, there is perhaps a principle here that, given a choice of taking the f- or h-pawn, one should take the h-pawn.
After 56. Rd3?? [Ed24-3.54], the play was 56. . . . Rxa7 57. f3+ Ke5 58. Bd1 Ra3 59. f4+ Ke4 60. Bc2 Ne3 61. Rxc3+ Nxc2+ 62. Rxc2 Rxg3 63. Kf2 Rf3+ (63. . . . Rg4 64. Rc4+ Kd5 65. Ra4 Rxh4, Fig. 3d) 64. Kg2 Rxf4, Fig. 3b. Our FinalGen computations ran from figures 3b and 3d. 65. . . . Rxh4 would have left a win with dtf = 44 moves and a first phase of nine moves leading to Fig. 3e. However, the chosen 64. . . . Rxf4 left a much more difficult win with dtf = 66 moves and a first phase of 37 moves in the line we followed leading to Fig. 3c. In both cases and particularly as played, White could have fought on for a while, hoping to be defending the usually drawn KRKRP(f)P(h) endgame (Müller and Konoval, 2016; Müller and Lamprecht, 2001) and/or hoping for the 50-move-draw.
Navrotescu–Shah: as played (a) 56w and (b) 65w, hypothetically leading with to (c) 103w after f4; the easier win (d) 66w after Rxh4 leading with to (e) 75w after h4.
Position 12: Vidit Gujrathi–L’Ami, NED–IND, 43rd Olympiad r10, Batumi 2018
Vidit Gujrathi–L’Ami: as played, (a) ‘won’ 60w, (b) drawn 61b and (c) 7m-draw 69b. In the chosen winning line by FinalGen, (d) 66. g5 and (e) 76. f5 are wiser and later advances than the played line.
At position 60w of Fig. 4a, FinalGen says White has a win in 33 moves. However, Vidit Gujrathi played 60. f5?? and the position is now drawn, a fact which totally escaped top engine Stockfish, kibitzing at the time (Chessbomb, 2018). Position 61b of Fig. 4b required 61. . . . Qd2 but 61. . . . Qc1? was played, and now FinalGen says ‘win or draw’. In fact, all lines seem to lead to a draw with pawns and evaluations stalled at 0.00, or with a 7-man draw as at position 69b as played, see Fig. 4c. An analysis engine recognising position-repetition and interfaced to Finalgen and 7-man EGTs would speed searching here. The winning optimal line from Fig. 4a is a lesson in festina lente, showing much later advances of the g-pawn as in Fig. 4d and the f-pawn as in Fig. 4e. Both had been providing some shelter from a rainstorm of Black checks until it abated.
We are not dependant on FinalGen for the next observations but it is worth noting how difficult this 6-man endgame is. On thirteen occasions, the most challenging play leaves value preserved by a unique move. The half-point separating Vidit Gujrathi’s win from L’Ami’s draw in fact transferred six times, with moves 71. . . . Kh8, 72. Qe5+, 76. . . . Qa3+, 82. Qe3, 85. . . . Kh8 and 87. h6. After 89. g5 and another 50 moves, the draw was claimed by L’Ami under FIDE Article 9.3.1.
Position 13: Tiviakov–Korsunsky, Frunze Open, 1989, 45w
The position of Fig. 5a was long thought to be a fortress, with the bishop on the a7-g1 diagonal blocking Kc6. Nevertheless, Hawkins (2012) first analysed this as a win with the help of 6-man EGTs. FinalGen provided confirmation and a clear, benchmark analysis (Müller and Haworth, 2013), fully considered in the first author’s latest work (Müller and Schaeffer, 2018). Here, we merely supply the played line and, for comparison, an line which assumes pawn-promotion is the main objective and minimaxes DTF, DTM and DTZ in that order.
The ‘computer line’ below highlights the intrinsic difficulty of this endgame. Certainly, Tiviakov would have found it hard to win this if Korsunsky had somehow found the best defence on each move. The surprisingly early pawn-offer, which cannot reasonably be refused, takes us into a dual between rook and bishop with the rook ultimately the victor.
Tiviakov–Korsunsky: positions in the played line, (a) 45w, (b) 62w, (c) 71b and (d) 79b. The FinalGen line chosen converts to KRPKBP at (e) position 51b.
Position 14: Leela Chess Zero–Ethereal, TCEC Cup 1, game 14, pos. 90b
The radical ‘neural network’ and rapidly improving chess engine ‘LC0’ was of major interest to the inaugural TCEC Cup audience (Haworth and Hernandez, 2019). Ethereal was also a dark horse, newly risen to the Premier Division of the TCEC league. The ‘best of eight’ rapid tempo match between them passed 4-4 en route to ‘best of two’ and featured unusually long contests, largely because LC0’s evaluations frustrated the TCEC adjudication rules. Game 14 was no exception.
This theoretically reached FinalGen territory at position 74w which requires 11TB for workspace so we quickly advance to position 94w of Fig. 6a. Removing the f- or g-pawns leaves a win with dtf = 29m/57p or 28m/55p respectively. Removing the four pawns leaves a draw. This suggests but does not guarantee that the 9-man position 94w is decisive. FinalGen proves that White did in fact have a win: dtf = 159 plies: the FIDE 50-move rule, with the advancing pawn on b4 and pawns to advance or capture, appears not to be an issue. Figs 6(b–d) mark successive phases of the endgame. The actual game was adjudicated by the ‘TCEC draw rule’ at move 175b, Fig. 6e.
LC0-Et: (a) 94w; FinalGen-optimal line (b) 105w before b6, (c) 7m 116w and (d) 6m 128b; (e) drawn 175b.
Position 15: Karpov–Kasparov, World Chess Ch. 1984, game 9, 66b
This is the ‘Fermat’s Last Theorem’ of endgame quests, a position that has inspired more incorrect arguments than any other, so many that the analysis of those arguments would be a separate and interesting thesis of its own. Fortunately, this saga is now at an end as we can definitively assess both the ‘ground truth’ and the false proofs with FinalGen.
The game is no less significant than the endgame. It saw Karpov go 4-0 up and preceded 17 draws in the controversial championship match eventually abandoned at 5-3 after five months and 48 games. The position of Fig. 7a has been an enduring mystery, visited many times over the years (Dvoretsky, 2008; Karolyi and Aplin, 2007; Marin, 2006; Müller, 2003, 2005; Müller and Konoval, 2018; Timman, 2007). Was the draw missed (Kasparov, 2007) with the move 66. . . . Bb7?
Karpov–Kasparov: (a) pos. 66b; drawing line (b) 73b and (c) 77w; in a better defence (d) 69w and (e) 74b.
FinalGen shows not only that the amazing defence 66. . . . Bh1 draws but that it is the only drawing move. Figs. 7b–c are taken from the drawing line. It also shows that the path to the winning line after 66. . . . Bb7?? is long and narrow with six ‘only’ moves for White. Figs. 7d–e are from a ‘best defence’ albeit losing line. After we had run this FinalGen computation in 2017, Jan Kominek pointed out that Roman Jiganchine (2015), whom we salute, had already got the same result. It is intriguing to think that there must be many other famous game and study positions (van der Heijden, 2017) whose FinalGen analyses have not been widely promulgated or even done.
Summary
FinalGen works beyond today’s endgame table limit of seven men, bringing definitive knowledge to the table, providing benchmark standards of play and resolving questions that have remained unanswered for as long as thirty years. In the context of current technology, it has untapped potential both as a working program and as a concept.
It is currently single-threaded but the principles of concurrent programming are well known and cpu chips continue to offer more cores and processing threads. PC B’s 4-lane PCIe v3.0 interface to its SSD provides over 1GB/s but PCIe v5.0 (PCI-SIG, 2019) in 16-lane mode promises to offer 16 times the bandwidth, effectively reducing disc-latency to zero. Romero (2012) provides code libraries to interface chess engines to FinalGen EGTs: we do not believe these have been used yet. FinalGen does not create value-only EGTs as de Man et al. (2018) do. Nor does it use two ideas exploited in Bleicher’s Freezer (Müller, 2005), namely trial-constraints on the chessmen and interfacing to precomputed EGTs. The full analyses (Müller and Haworth, 2019) show how truncated FinalGen’s searches would be if 7-man EGTs could be consulted. These considerations suggest that the efficient creation of FinalGen-type EGTs could be a part of chess engines’ runtime calculations.
More immediately, a growing FinalGen community, centred around the forum (Romero, 2012) could evolve and share future results. Endgame tables now give instant answers for 7-man positions but FinalGen can address many positions with eight men or more. The examples here will hopefully inspire others to show what is possible in creating further unarguable insights into the game of chess.
Footnotes
Some logistics details on working with FinalGen and ‘syzygy’ EGTs
One caveat: FinalGen does not reliably pick up the ‘side to move’ when importing a position. Finalgen can create files larger than the 4GiB limit of the venerable FAT32 file system. Media can be reformatted to support NTFS which allows files to ∼8PiB.
An SSD’s lifetime is determined by the intensity of writing activity. It is therefore worth considering whether or not one should compress SSD-files from inception. Windows’ NTFS compression uses the ‘IEEE Milestone’ LZ77/LZ1 algorithm (Ziv and Lempel, 1977). Being linear and highly efficient, LZ77 requires minimal rewriting to disc. Given that NTFS is achieving 29% compression on ‘Normal mode’ (and 39% compression on ‘SdF’) files, there is an argument for compressing FinalGen files on SSDs from inception. Further, doing so with HDDs cuts disc-latency time. On PC B, the computation overhead was about 40% in ‘Normal Mode’ and 7% in SfD mode, arguably worthwhile.
References
1.
Chessbase (2018). https://tinyurl.com/icgaj027. Endgame Magic 110 with Karsten Müller and Sagar Shah. Navrotescu–Shah game and ending.
2.
Chessbomb (2018). https://tinyurl.com/icgaj021. Vidit Gujrathi–L’Ami, Netherlands–India, round 10 of the 43rd Chess Olympiad, Batumi.
3.
de Man, R., Fiekas, N. & Guo, B. (2018). https://tinyurl.com/icga007. Fiekas’ interface to ‘Syzygy formatted’ de Man sub-7-man and Guo 7-man DTZ50″ EGTs.
4.
Dvoretsky, M. (2008). Dvoretsky’s Analytical Manual. Russell Enterprises.
5.
Guo, B. (2018). ftp://ftp.chessdb.cn/pub/syzygy. EGT download site: sub-8-man DTZ50″ EGTs.
6.
Hawkins, J. (2012). Amateur to IM – Proven Ideas and Training Methods. Mongoose Press.
7.
Haworth, G.McC. & Hernandez, N. (2019). The TCEC Cup 1. Submitted to the ICGA Journal.
8.
Jiganchine, R. (2015). https://tinyurl.com/icgaj018. Computer precision in a historic endgame: Karpov–Kasparov, 1984, game 9.
9.
Karolyi, T. & Aplin, N. (2007). Endgame Virtuoso: Anatoly Karpov. New in Chess.
10.
Kasparov, G. (2007). On My Great Predecessors (Vol. 5). Everyman.
11.
Lomonosov (2012). http://tb7.chessok.com/probe. Query service to sub-8-man DTM EGTs.
12.
Marin, M. (2006). Learn from the Legends: Chess Champions at Their Best. Quality Chess Europe.
13.
Müller, K. (2003). Endgame Corner 43, September. On Karpov–Kasparov.
14.
Müller, K. (2005). Endgame Corner 50 and 55, Feb. and July. Freezer, and Karpov–Kasparov.
15.
Müller, K. & Haworth, G. (2013). Rook versus Bishop. ICGA Journal, 36(4), 195–202. ISSN: 13896911. http://centaur.reading.ac.uk/36189/. doi:10.3233/ICG-2013-36402.
16.
Müller, K. & Haworth, G.McC. (2019). http://centaur.reading.ac.uk/80432/. This note plus data, pgn files and an ancillary text with the full analyses and annotations by FinalGen and the authors.
17.
Müller, K. & Konoval, Y. (2016). Understanding Rook Endgames, esp. §2.2.2. Gambit.
18.
Müller, K. & Konoval, Y. (2018). Understanding Minor Piece Endgames. Russell Enterprises. ISBN: 978-1-941270-78-3(print), 978-1-941270-79-0 (eBook).
19.
Müller, K. & Lamprecht, F. (2001). Fundamental Chess Endings. esp. §6. 2. A2d. Gambit.
20.
Müller, K. & Schaeffer, J. (2018). Man vs. Machine: Challenging Human Supremacy at Chess. Russell Enterprises. ISBN: 978-1941270967.
21.
PCI-SIG (2019). https://pcisig.com/. The PCI Express Special Interest Group.
22.
Romero, P.P. (2012). https://tinyurl.com/icga013. FinalGen: Tutorial, download and forum.
23.
Rusz, A. (2018). Private communication confirming FinalGen ‘win or draw’ results.
24.
Timman, J. (2007). Power Chess with Pieces. New in Chess.
25.
van der Heijden, H. (2017). http://hhdbv.nl/. Endgame Study Database V.
26.
Ziv, J. & Lempel, A. (1977). A universal algorithm for sequential data compression. IEEE Trans. on Information Theory, 23(3), 337–343. doi:10.1109/TIT.1977.1055714.
