Long Work In Progress

It has been more or less 6 months since the last Persy release, and usually this is the time to do a new release, but as time of writing there is not much to release yet, so I'm writing this blog post to give a status update on the current working in progress.

Status

Looking to the master branch of the project there are not many new commits, at the moment of writing there are only 34 commits in the master more than what has been released in previous hotfixes, this include one slim new feature that allow to inspect some internal state of Persy mainly for debugging purpose.

The commits in master is everything that is good enough to ship, which is not much now, but a lot of more work has been done that is not yet ready to be released, here the details:

Limit of sequential delete of key in index

This is a bug that has been there for a long while (Issue), for correct this problem I had to refactor the logic for locking the index and then restructure the whole update process of the index, this resulted in a massive work, that is not yet done, you can see the work in progress in this branch which at time of writing it does have all the suites of tests in the base repository passing, but I could reproduce some errors running 20 threads concurrently that remove and add more than 9000 keys in the same range of values, try to fix this concurrency problem is taking quite long, and I may need to write some more specific test for check some state transitions that so far I could only trigger with big concurrent tests.

I may need to introduce some fuzzy testing at some point, but fuzzy testing is good mainly to find some logic flow that is not covered correctly, which I kind of have already with the multi thread test, trying first to wrap my head around this specific flow and try produce unit test for the specific missing cases.

Crash Resilience

I did notice in some cases that if process doing writes in Persy crash, the file went in a sort of corrupted state, this often happen with a double crash, like first a kill of a writing process and then another failure on recovery, I'm working to produce some testing that run and kill process to collect files that are in corrupted state or pre-corrupted state to try to trim down the cases where this happens, the first implementation is published here.

Tracing feature

Another optional feature I'm working on is the integration of tracing, I did a small adapter implementation to be able to turn it on/off on need, and I'm slowly adding spans around the code to be able to track running operations, this is in yet in early stage and the code can be found in this branch

Organization

Recently the number of I'm implementing projects related to Persy is growing more, so now there is new organization on GitLab where all the Persy related projects and soon enough Persy itself will be moved in the organization here is the link to the organization

That's all for now I home I can come soon with some a new release that fix some hard issues