Persy 0.10

Has been quite some time since the last release and this release pack quite a lot of changes in it, not exactly following the planned path, but still a nice set of changes.

What is new in Persy 0.10

Fixes

There have been a few fixes here and there, the one worth mention are:

  • The support of ignoring newly created records from snapshot based scans
  • Multi state tracking of transaction cleanups this avoid disc leaks in case of crash and recovery, is still possible that some disc leak happen in some corner case of crash but would be quite rare

Improvements

In term of improvements we have a bigger set of changes

Reuse of address structural pages

One of the Persy structures called address didn't support yet the reuse of pages that become empty, since this version this is implemented so all the internal structures of Persy now support disc space reuse, still though the return of space to the system trimming the file size has quite big limitations, will try to address this in future.

Add support of in memory only

Since this version is possible to create a Persy instance with in memory "storage", this means that all the logic of Persy will be backed by a Vec<u8> instead of a file, this is useful for tests and for all the cases where a temporary instance is needed, obviously this mode is faster than the mode that use the disc but because of all the structures of Persy this is not as fast as the systems designed to be only in memory.

Here the simple way to create a memory instance:

let persy = OpenOptions::new().memory()?;
...

Add support of background disc sync

As optional feature in this release has been included the support of disc sync(fsync) in background thread, so now when a transaction begin you can select if wait for disc sync or return acknowledge before the transaction is actually flushed to the disc, this option just schedule the sync on a background thread and return the control immediately instead of execute the sync in the current thread. The main advantage background sync is that single thread operations have much lower latency, for example with this option I could run few thousand transaction per second with one thread, instead of a few hundred per second.
The trade off of this flag is that in case of a full system crash, the last transaction may not be present in the database, a normal process crash should not create any issue because all the operations have been already sent to the OS before return the control, only if the OS will crash there is the possibility of some missing transactions, so the case should be quite rare especially on devices with a battery.
The flag do not change any structural flow so cannot produce a corrupted database.

Here an example:

persy={version="0.9", features=["background_ops"]}
let persy: Persy = //// .....

let mut tx = persy.begin_with(TransactionConfig::new().set_background_sync(true))?;
// ...
tx.prepare_commit()?.commit()?;

The main reason why this feature is optional is the need of a background thread, libraries should not start threads so for guarantee that Persy do no require threads this feature is behind a feature gate.

API Clean

The APIs have been cleanup, all the old APIs are still there but deprecated, there is a reference to the replacement API in the deprecation message, the new APIs will be the definitive for the 1.0 and all the deprecated APIs will be remove in the next release.

Future developments

There are a few things that are still needed to get done for a 1.0 the most important change will be starting to guarantee disc compatibility across versions, there are some other small changes and optimization that can land before a 1.0 mainly doing all the disc optimization before freeze, but the main API and the main structures are complete today so no big changes expected in the next releases, especially in the APIs.