Research cloud-based storage backends

[taoeffect]

Problem

In some apps the amount of space on the VPS might be insufficient to host the app.

Solution

We'd want a backend that would store most data in some massive cloud storage thing, and then have a local cache on the VPS that it pulls data into it. It would then transparent read and write between the cache and the cloud storage.

I'm using this issue to keep track of potential solutions for this.

• https://github.com/huggingface/hf-mount
• https://github.com/Barre/ZeroFS
• https://github.com/archiecobbs/s3backer
• https://github.com/s3ql/s3ql/
• https://github.com/juicedata/juicefs
• https://rclone.org/commands/rclone_mount/
• https://github.com/kahing/goofys
• https://github.com/s3fs-fuse/s3fs-fuse
• https://github.com/russellromney/turbolite (hn)

Feel free to suggest others to research!

[corrideat]

My experience with this (a few years back, so may not apply entirely) is that 'mounting' an S3-drive or similar doesn't work as reliably as one would wish. Based on that, creating an S3-based backend might be a better approach (and the S3 API seems to be what most cloud providers offer).

Also, even if the FS mount approach worked reliably, it seems unnecessary to have to go through the POSIX abstraction layers.

Something to keep in mind for the accounting part is that many cloud providers charge for egress data (in addition to storage), so the accounting module might need to be updated too.