Persy.rs

Resources

Persy 1.5

Has been quite a long time since the last minor release (more or less 1 year), here we are with a new minor release!! Between the last minor and this there have been 7 patch releases, that solved a bunch of issue reported by the users, but some problems required major re-work and could not be fixed in a patch release, so changes are included in this release. Checking the diff with the previous release on git, the stats are 47 files changed, 3552 insertions(+), 903 deletions(-), this amount of changes is comparable with the previous minor releases so everything looks fairly normal, even though the complexity of some changes has been quite higher, so it took additional time to get done.

Support

If you want to support my work on Persy feel free to sponsor me

Changes

The set of changes covers mainly the index logic, there are three major set of changes, in transaction index changes, tree rebalance and locking logic, Inspect APIs.

TLDR;

New Container of transaction index changes, produce visible speedup in compilations times
Fix rebalancing bug
New experimental inspect APIs

Transaction Index Changes

There have been a refactor (so not changes on public logic) on the way index changes are kept in the transaction, before there was a "sort of map" that kept the values using enums with all the supported types, this implementation has static dispatch and cause the "ahead of time" compilation of the index implementation, for all the combination of keys and value types that are supported by Persy (I counted somewhat like ~250 combinations), this caused some really slow compilation time, I replaced this structure with a structure that kept the index changes in a dynamic structure based std::any::Any and the compilation time is decreased of a large margin. I do actually have both implementation in the code with the static dispatch behind a feature flag, so I can easily compare the compilation times, and this is the result:

dynamic impl:


Base build: 
[unoptimized + debuginfo] target(s) in 14.96s
doc tests:
test result: ok. 60 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.02s

static impl:


Base build: 
[unoptimized + debuginfo] target(s) in 41.18s
doc tests:
test result: ok. 60 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 19.86s

So now you can enjoy faster compilation times, as well will be easier to add types in indexes in future release with no impact on compilation performances.

Index Rebalancing and locking logic

In the previous minor there was a bug when deleting many sequential keys from a single index, in a single transaction, this was because the rebalance logic didn't work well when going over two level of pages, to fix that I had to refactor the locking logic to make sure I was locking all the correct pages in the correct order, and then change it, this took quite a lot of time to get done, I can still reproduce some issues in high concurrency, with ~30 threads inserting and deleting enough sequential keys to require a multilevel rebalance, this bug is rare and I could include some mitigations for it, so I'm happy to release these changes even though are not as perfect as I wish them to be, in any case is a big improvement compared the previous release.

Experimental Inspect API

I did struggle a lot to debug problems on indexes and other APIs, especially when I could not access the database files directly, so I did add some API, that allow me to extract some information without the need to have the files myself, these are somewhat public APIs, but I do not plan to make it stable soon and for sure not in this release, so I included them behind the feature flag experimental_inspect and now on everything that is not considered yet stable will be behind an experimental_* feature flag.

I will be building debugging tools based on these APIs in the future, I will keep you posted on this.

Future

Since 1.0 my focus is on stabilization and bug reports, so I keep features as secondary, I do have some branches that experiment on: tracing support, different index APIs, no copy record data access, It will depend on how many bugs and how much time I do have if they land in the next release