Transactional Disc Safety

Persy being a transactional(ACID) storage have to guarantee when a transaction has been confirmed to be written successfully even if there is a complete software crash the transaction is written, in the opposite if there is a software crash before the transaction is fully completed the data should be rollback-ed to the previous state, and more important does not matter when the system crash a valid state of the data should be accessible after a recovery, should never happen that a crash make the existing data not accessible.

To guarantee this property since the first release Persy use a copy on write approach on the user data, in more detail, Persy for each segment has an addressing structure that for each record keep where the data is located. In case of update of a record the new data is written on a new location of the disc (in the specific of the file), and then the address of this location is written in a log, as soon a final commit is requested Persy make sure that all the data written until this point and the log data is flushed on the disc, when this is done the new locations for the record are written in the addressing structure, and even if there is a software crash at this point is possible to recover the new locations from the log and re-apply them to the addressing structure.

This approach works really well for the user data, but Persy is a single file storage, so inside a single file there are multiple structure ( check here for more datils ), each of them having it's own metadata written on the disc, even if this metadata is not written as often as user data, there is a chance in case of software crash that one of this structures are written partially on the disc, making the whole data not accessible, even the copy on write logic would not help here because that logic is actually implemented on top of this structures.
Since 0.5 though as been introduced a new format for this structures that take inspiration from the copy on write logic, now each structure has double of the space it require, and the metadata is written twice on the disc, with a version to determinate what is the more recent, and only one of this spaces can be written at any time, so even if there is a software crash while flushing the metadata it can be detected on restore just checking a checksumm a valid (maybe a previous) version of metadata will be every-time present on the disc. This new implementation guarantee that will be every-time possible to open a Persy file even in case of a software crash, even though in case of crash some newly allocated portions of the file may became stale creating a minor quite rare disc leak.

With this solution Persy now guarantee that can be accessed at any time even in case of crash, the only point still open is the possibility of some file systems (like network file systems) to fail to sync data on the disc, more investigation need to be done on how to detect this using the rust file API, in any case when the detection is solved is easy track at high level the last data that was supposed to be sync and re-write & re-sync it.


I refer every-tyme to software crash, because hardware crash need a completely different set of solution that are not and probably will never be implemented in Persy.