Publish a "snapshot version support" document

[jorisdral]

lsm-tree has a versioned on-disk format for snapshots. lsm-tree is backwards compatible with older snapshot versions. Forwards compatibility is not supported at all. lsm-tree committed to being backwards compatible with at least one snapshot version at all time, but it is not its goal to be backwards compatible indefinitely.

Users should have access to a document that describes lsm-tree's procedure of adding new snapshot versions and dropping old snapshot versions. Moreover, the document should describe how users are affected by such snapshot version changes and how they should structure/modify their projects accordingly. Ideally the document would answer:

• How are users notified of new snapshot versions?
  • For example: a major lsm-tree version bump and an entry in the changelog
• How should users structure their projects to account for future snapshot versions?
  • For example: they should have tight enough bounds on lsm-tree versions so that all those lsm-tree versions produce the same snapshot version. Or, if they don't have such tight bounds, then they should implement a recovery strategy.
• How should users migrate from old snapshot versions to new ones?
  • For example: open an old snapshot with a newer lsm-tree version, and then write it back to disk with the new format
• How are users notified of dropped old snapshot versions?
  • @wenkokke suggested using a MAJOR.MAJOR package version bump to signal dropping support for older snapshot versions. For example, we could bump up from version 1.x to 2.0.
• how long should users expect versions to be supported?
  • For example: at least a year?

[jorisdral]

Something I also realised is this: adding a new snapshot version can have quite far reaching consequences for packages that (transitively) depend on lsm-tree. If the dependency bounds are not set right, then small changes in the cabal build plan can cause the lsm-tree dependency to be downgraded to an older packages version with an older snapshot version. This makes snapshots that were created before then unusable (unless there is recovery logic in place).

[wenkokke]

How should users migrate from old snapshot versions to new ones?

Would it be possible to release a series of lsm-tree-upgrade-vX-to-vY packages that convert database formats one step at a time? Those wouldn't necessarily need to be maintained much or at all – especially if you keep them as lightweight as possible – but they'd need to be separate packages to allow using multiple ones at the same time. Not requesting this as a feature, just think it'd be a good thing to have to make backwards compatibility possible without an indefinite maintenance burden.

[jorisdral]

If you open an old snapshot version using a newer version of lsm-tree then you automatically get the newest format as soon as you write a new snapshot, so I suppose a converter would mostly be useful when we decide to drop older snapshot versions and maybe people still have an old snapshot version lying around that the new lsm-tree version does not support anymore. Do you have another use case in mind?

Bottom line is I think we could take it into account for the version support document, but I wouldn't want to commit to releasing such packages just yet

[jorisdral]

Or are you suggesting lsm-tree depends on these lsm-tree-convert packages rather than users of lsm-tree? Because that sounds a little more convincing because it makes the snapshot decoders "compositional" in a way, though at the cost of code duplication across the lsm-tree-convert packages

[wenkokke]

Or are you suggesting lsm-tree depends on these lsm-tree-convert packages rather than users of lsm-tree? Because that sounds a little more convincing because it makes the snapshot decoders "compositional" in a way, though at the cost of code duplication across the lsm-tree-convert packages

No, no, absolutely not. I'm suggesting making binaries available that perform one step of upgrade by pinning exact lsm-tree versions. As you said, if you're always backwards compatible one format version, then you can quite easily make these binaries available to chain longer upgrades.

[wenkokke]

Here's my suggestion:

Whenever you change the snapshot format from X to Y, you copy its loadSnapshot logic function into a compatibility modules (e.g., Database.LSMTree.Internal.Snapshot.VX) with a datatype that represents the old in-memory representation and write a short function that converts the in-memory representation from X to Y. This function does not have to be efficient. It should probably be linear, but simply reinserting every entry is probably an acceptable start. In the main loadSnapshot function, you read the version, call the appropriate loadSnapshotVX function and chain the conversion functions until you arrive at the most up-to-date in-memory representation.

The guarantees that lsm-tree should offer are then:

1. The lsm-tree-X.Y package can read any snapshot produced by lsm-tree-Z.W where X == Z && Y >= W or X == Z+1, which is achieved as above.
2. For any two versions lsm-tree-X and lsm-tree-Z where X == Z+1 there exists a package lsm-tree-upgrade-vZ-to-vX.0 with an executable of the same name that converts snapshots of version Z to version X.0. This should be easy to implement given (1), i.e., pin the final lsm-tree-Z and import/export the snapshot.

These two constraints should make it possible to upgrade a database from any past lsm-tree version to the latest version. (It might not make it easy, given that you'd need older GHC versions to build the older versions, but I don't suggest putting any work into maintaining older versions. Users will likely only need to run the update once, so they can figure out how to Docker it. You could supply a Dockerfile or container if you'd like to make it easier.)

Moreover, maintaining these constraints should enable you to give informative ErrSnapshotVersionUnsupported messages that tell users about which lsm-tree version is needed to read this database format, regardless of whether it's an older or newer format than expected, and, if it's an older format, tell you exactly how to upgrade it.

I don't suggest putting any effort into downgrading formats.

It might be tempting to say, in (1), that if X == Z+1 then only the final Z.W version is supported, i.e., a major-major release breaks compatibility with all but the previous major-minor release, but this makes (2) significantly harder to implement, as only lsm-tree-Z.W will know how to upgrade the database revisions made between lsm-tree-Z.0 and lsm-tree-Z.W, so you'll need two executables lsm-tree-upgrade-vZ-to-vZ.W and lsm-tree-upgrade-vZ.W-to-vX.0. Might be worth a shot, since it might alleviate maintenance burden, but definitely more annoying for users.

The schema described above does allow you to break backwards compatibility entirely, but only permits you to drop support after one major-major release.