jja: convert CTG books to PolyGlot format (and more!)

[alip]

I want to announce jja, a command line utility to interact with various chess file formats. It is still in its very early stages of development. The initial intention of the author was to convert their opening books which were saved with ChessBase's proprietary CTG format to the free and open PolyGlot format. Overtime they intend to add support for other chess file formats (cbh, epd, pgn, si4, si5 and so on).

home: https://crates.io/crates/jja
git: https://git.sr.ht/~alip/jja

Installation is as simple as "cargo install jja" on Linux.
This is free software, licensed under GPL-3.0-or-later.

# Notes about CTG book conversion
This is currently somewhat primitive. jja uses the NAGs and performance rating in the CTG file to determine the weight of the corresponding PolyGlot move. Overtime I want to make this more configurable and potentially include more ctg fields into calculation.

I hope you find it useful, please let me know what you think.

[alip]

As of version 0.3.0, jja supports reading/querying:

PolyGlot, aka bin
Arena, aka abk
ChessBase, aka ctg
ChessMaster, aka obk (version 1 and 2, w\o,with text notes)

opening book files, wheras it supports writing/converting to:

PolyGlot, aka bin

opening book files.

During opening book conversion, jja uses the information provided in various input opening book formats to come up with a move weight which accompanies the move in the PolyGlot opening file. jja also writes some custom numbers in the learn field, such as NAGs during ctg conversion or priority during abk conversion. You may disable this custom usage using --no-learn as it may confuse other software making use of this field.

[alip]

jja-0.3.1 is released today. As of this version jja can write Arena, aka `abk` books. Note, Arena, aka abk, opening book file writing support is only supported from Chessbase, aka ctg books. Use the command line flags --author, --comment, --probability-priority, --probability-games, --probability-win-percent to configure ABK header metadata. Game statistics (minimum number of games/wins, win percentages for both sides) are managed automatically by jja.

In-place editing for Arena opening books is also possible using -i, --in-place=SUFFIX command line option. Conversion from PolyGlot, aka bin, and ChessMaster, aka obk opening books to Arena, aka abk opening book files is planned for a future release.

Install

To compile from source, use

Code: Select all

cargo install jja

. This requires the rust toolchain to be installed.

As an alternative, static builds of jja are hosted on chesswob.org. These versions are signed by GnuPG, using key D076A377FB27DE70. To install, acquire the latest version from chesswob.org, verify the checksum and the GnuPG signature:

Code: Select all

$> curl https://keybase.io/alip/pgp_keys.asc | gpg --import
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13292  100 13292    0     0  13535      0 --:--:-- --:--:-- --:--:-- 26584
gpg: key D076A377FB27DE70: public key "Ali Polatel (Caissa AI) <alip@caissa.ai>" imported
gpg: Total number processed: 1
gpg:               imported: 1
$> for f in jja{,.sha512sum,.sha512sum.asc}; do wget -q https://chesswob.org/jja/${f}; done
$> gpg --verify jja.sha512sum.asc jja.sha512sum
gpg: Signature made Sun Mar 19 20:52:41 2023 CET
gpg:                using RSA key 5DF763560390A149AC6C14C7D076A377FB27DE70
gpg: Good signature from "Ali Polatel (Caissa AI) ...
$> sha512sum -c jja.sha512sum
jja: OK
$> sudo install -m755 jja /usr/local/bin


More from the ChangeLog:

edit learned to calculate & write ABK header game statistics fields
edit learned to convert CTG book files to ABK book files
edit learned --author and --comment to specify metadata for Arena opening books
edit can edit Arena opening book (abk) files in-place with -i, --in-place=SUFFIX
support for writing Arena (abk) opening books
open learned to wrap long ECO opening lines into multiple lines
find no longer panics on some abk books with entries having invalid uci
edit takes move priority into account for weight on abk to bin conversion
match learned --move-selection={best_move,uniform_random,weighted_random} to pick move selection algoritm for book moves
fix castle decoding on polyglot read/query
fix error return when no positions found in abk, obk and ctg find
fix promotion handling in ctg edit
many improvements to ctg find (move coloring & sorting, average statistics)
new merge command to merge two PolyGlot opening books
new match command to arrange book matches with random playouts

[alip]

jja-0.3.2 is released today. As of this version jja cross-compiles successfully for Windows and we're distributing 64-bit windows executables for jja too.

Download:
- Linux: jja-0.3.2.bin, sha512sum, sha512sum.asc
- Windows: jja-0.3.2.exe, sha512sum, sha512sum.asc

The Windows version is only briefly tested under Wine. Please test under Windows if you can and let me know if there're any issues.
Finally here is an asciicast of jja in action:


More from the ChangeLog:

edit writes the name of the user and jja's version as comment to Arena opening book metadata on ChessBase to Arena opening book conversion, override with --author, --comment
edit displays a unicode chess board in edit tempfile
fix book traversal on Arena opening book to PolyGlot conversion
enable ansi colors when running on windows terminal
drop unixisms, cross-compiles for windows
fix yet another bug with castle decoding on polyglot read/query
edit no longer tries to spawn the default editor if standard output is not a TTY
fix find and edit for Arena opening book reading, move selection is on par with the Arena GUI
new hash command to calculate the Zobrist hash of the given position
fix infinite loop while converting some big Arena opening book files
improve hashing performance by avoiding double hashing using a hasher builder for Zobrist hashes
improve hashing performance using shakmaty crate's Zobrist Hashing implementation rather than the internal one.

[alip]

We're thrilled to announce the release of jja 0.4.0, an update to our versatile chess file format tool. The latest version introduces the ability to create PolyGlot books from PGN files, including compressed PGNs. It also features improved performance and memory management for handling large PGN files and the ability to filter moves using Filter Expressions for creating specialized opening books.

In addition to these improvements, we've generated five PolyGlot books for the community, which are available for download under the Creative Commons CC0 license. You can use them for research, commercial purposes, publication, or anything else you'd like.

To get started with jja 0.4.0, follow the installation instructions inthe original blog post. We look forward to your feedback and are excited to see how these new features will help chess programmers elevate their game. Gens una sumus!

[alip]

We're excited to announce that JJA has reached another milestone with the release of version 0.5.0! Following our previous version 0.4.0, this latest update brings substantial enhancements, critical fixes, and notable new features that are sure to make your chess analysis even more powerful. Read more about it in the blog post here.

Download

- Windows: jja-0.5.0.exe, sha512sum, signature
- Linux-Glibc: jja-0.5.0-glibc.bin, sha512sum, signature
- Linux-Musl: jja-0.5.0-musl.bin, sha512sum, signature

To build from source, use

Code: Select all

cargo install jja

ChangeLog

0.5.0

important fix, for encoding of castling in polyglot books. Previously, we encoded castling as e1g1, e1c1, e8g8, and e8c8, whereas the correct encoding is e1h1, e1a1, e8h8, and e8a8.
when printing version with --version, prefer git version over package version for git builds
upgrade clap crate from 4.2 to 4.3
upgrade ctrlc crate from 3.2 to 3.3
upgrade rocksdb crate to 0.21.0 which bundles RocksDB-8.1.1
document all the public code and enable the lint #[deny(missing_docs)]
edit learned to export opening books in PGN format, use an output file with pgn extension to export an opening book to a PGN
add back the build dependency upon the built crate
find learned -l <max-ply>, --line=<max-ply> to display lines from the opening book as a table of opening variations reverse-sorted by cumulative weight
make --min-score now accepts floating point values as argument rather than an unsigned 64-bit integer
make learned --win-factor, --draw-factor, and --loss-factor to specify respective factors during score calculation, the defaults, 2, 1, and 0 respectively, resembles the original polyglot tool
find --tree=<max-ply> no longer panics on broken pipe, so it's more convenient to use with a pager
make learned -p, --min-pieces to specify the minimum number of pieces on the board for a position to be included in the book, defaults to 8
merge learned -c, --weight-cutoff to specify the minimum weight of entries to be included in the book
merge learned about merge strategies avg and wavg to merge using average weight or weighted average weight respectively; the weighted averages should be specified for wavg using -w, --weight1, and -W, --weight2
in-place editing of polyglot files allows editing empty books which makes it practical to create polyglot book from scratch
merge learned about merge strategies max, min, ours, and sum, default is sum which adds together move weights, max picks the one with the maximum weight, min picks up the one with the minimum weight, and ours always picks the entries from the first book
translate the README to German language
turn i18n support into a feature which defaults to on. For static linking this must be disabled as embed makes use of proc macros.
drop depedency on unused libc crate

0.4.1

fix docs.rs build
update shakmaty, and pgn-reader creates
add initial translation to Turkish language
add initial translation to German language
use gettext for i18n