Persy 1.3

Few months after the 1.2 here we are with 1.3 release, this includes a small set of improvements mainly in stability and maintainability, most of the work on this release was around refactors and fix some durability issues that lead to changes and disk format, but let's go through the whole list of changes:

Changes

Internal Refactor

This does not have any effect on the public API but has been the biggest set of changes of this release so it's worth a mention, some big modules have been split in multiple files, and as well some test modules were moved to specific files, this should also improve a bit the compile time speed.

Durability Fixes and Disk format change

The performance improvements in 1.2 introduced some durability issue after crash, hot-fixes were done in the 1.2.x series to mitigate the problem, but a more structural change was needed to solve completely the problem.

The main issue was related to the way free pages were tracked and re-used: the structure for track free pages until Persy 1.2 was a LIFO (Last In First Out) linked list. Together with concurrent transactions, this created some problems, a page could be re-used by a transaction before the cleanup of the transaction that freed it was finished.

This could cause problems during a crash recovery, the specific case of re-used page concurrently has been handled now correctly in the recover process, but in order to reduce the amount of pages re-used concurrently, the structure to handle the free pages has been turned into a FIFO (First In First Out) and this structure being persistent means this change affects disk compatibility.

Files created with Persy <1.2 will be converted automatically to the new structure when opened with Persy +1.3. Files created with Persy 1.3+ are not retrocompatible, therefore opening them with Persy 1.2 will fail.

In any case: before upgrading Persy, backup your files!

Record and Key/Values Limits

Data size for record and index key/values has been limited artificially to ~500MB for record and 500KB for index keys/values. The reason for this limit is to avoid problems with other tools and structures afterwards, like export/import of a record bigger than 500MB, it could create memory and or encoding issues.

This change has introduced a new entry in the errors for insert/update of records and put of a key value and technically is a breaking API compatibility in terms of errors, so please check your code when upgrading to Persy 1.3.

Recover process fix

There was a bug in the recovery process of free pages listed before, in some case some transaction were listed duplicated in the recovery process as well. This is now fixed.

Future

After this version, the next one will follow the same path: bugs fixes and possibly small new features. Two features that may or may not land in the next release are entire file encryption support and composite-key indexes. These features haven't yet been fully designed so is not guaranteed they will land.