Persy 1.4

Hello everyone! Six months after the 1.3 release here we are with a new shining 1.4, mostly a bugfix release with no new features.

This release include ~115 commits, for a total of: 54 files changed, 3705 insertions(+), 2938 deletions(-) which amounts to more or less 15/20% of the Persy code touched in this release. Anyway, let's go through what happened in these 6 months:

Changes

Refactor

The first set of changes are refactors aimed to simplify the project structure, remove some redundant code and split tests into specific modules hoping to improve a bit compilation times. Most diff changes come from this refactor.

Fix Snapshot Releases

Persy uses snapshots as public and private APIs. Snapshots allow reading and writing in parallel and thanks to some users reports some bugs were squashed that were causing the journal to grow a lot, and with it the file size and the time to recover when re-opening the database. This was fixed by implementing the Rust Drop trait, so compared to the previous versions, file size and re-open operations should be smaller and quicker.

Fix recover of free page list

Users have also reported issues when recovering from a crash. Some missing code branches in the free list recovery would cause spurious panics here and there on execution after a successful recover, this is now fixed.

Additional journal entry for rollback recover

Persy transactions are atomic and re-executable, meaning that when a transaction reaches the prepared state and then progresses to be committed, it can be safely re-executed even if the commit operations fail somewhere.

Now, when recovering after a crash, the user can choose to rollback a transaction that during normal operation was going to progress to commit. But in case of a database crash, changes that applied before crash by the commit now need to be reverted when recovering the database.

The journal, however, was missing these information, so the previous state was lost and not recovered after the crash. Persy 1.4 adds a new entry in the journal to handle the rollback of transactions that were going to be committed before closing (or crashing!).

This additional journal entry makes previous versions of Persy unable to read databases created with Persy >=1.4 (Persy 1.4 is of course retro-compatible with previous databases).

Fixes in journal read after crash

Database write operations can be interrupted at any time (say, the process dies or the hardware crashes) so journal pages could end up being only partially written. What to do in this case? Well, these pages are corrupted and not valid anyway so when reading back the journal, from Persy 1.4 the read is hardened and will skip these broken pages and will consider recoverable only up to the previous valid page. Users reports were instrumental to identify and fix this problem.

New Error on prepare transaction for concurrent create of indexes and segments

Previously, when two transactions tried to create the same segment or index and one of the two transactions fails, transactions were somewhat merged causing concurrency errors. Starting from this version, merging these transactions is not attempted anymore.

Recover process Memory Optimizations

Recovering a journal in general should not have too many transactions because they are clean as soon as no-one is reading old data anymore. Starting from this version though, memory usage used during a recovery is reduced by freeing the temporary data of all the fully completed transactions, which improve the open performances for the cases were the journal has a lot of transactions.

Fix Range Operations

Index range operations were returning wrong results when run with changes in the current transaction, this is now fixed.

Transaction id limit

Starting with Persy 1.4 the transaction id has been limited to 512 bytes to avoid possible failures with values not fitting underlying data structures.

Future

This release was not planned (who plans for bugs, anyway 😅). I may not lay out big plans for the short-term future though there are quite a few new features that I would love to include, along with some more cleanup and performance tuning. The goal for now is to stabilize everything and improve maintainability.