Implement a process that safely uploads images to an S3 bucket

[mattaereal]

We have an S3 bucket to store files, particularly images, so we avoid adding them to GitHub unnecessarily.

What I am thinking right now, is creating some sort of workflow, or integration, that upon sharing links to a folder or files, if a maintainer approves the workflow, it will upload them all to the S3 bucket and return the hosted files with unique IDs.

There are plenty of temporary online free hosting files like:
https://www.file.io/
https://tmpfiles.org/
https://file.kiwi/

[Robert-MacWha]

We could do it via github comments if we wanted to make it really simple for contributors lol -

Like people upload images into a PR thread in github, and when that PR is approved it's scraped for any images and they're uploaded to the bucket. Since admins can delete messages it'd be easy to filter them.

Silly idea :)

[mattaereal]

Not silly. Can you fully describe all the technical aspects for this idea? You mean uploading to user-attachments's assets, and leaving some sort of placeholder for the potential name that they'll use in advance in their commits?

[Robert-MacWha]

Looking into this a little more, I think it might not be as possible. Github places auth restrictions on accessing resources, so I think even if we stored images in the PRs we'd also need to inject some auth tokens at runtime. Possible, but probably more annoying than just a public s3.

A simpler alternatively: the average 1080p webp file is like 0.8mb, and I think Github places a soft repo limit at 5gb. That's easily several thousand images. We could always just run a preprocessing action in PRs to develop / main, scanning for any optimized images and converting them. I've actually done that exact thing for my own website in the past, it's pretty trivial. I'm not sure how this would relate with cloudflare / the website host (since we might run into fs limits on their side?) but even if we did images inside github repos don't have the same auth restrictions as images in threads, so they could just be stripped from the build.

For videos, we could always just restrict "point to a video from youtube, unlisted if you'd like, instead of embedding in the repo". Which is probably best practice anyways.

Essentially you just use this action (https://github.com/calibreapp/image-actions) and then grep for the old image names & replace them with the new ones.

[Robert-MacWha]

To summarize:

When a steward wants to add a new image, they will:

1. add the image to a standard assets dir in the repo
2. Reference the image from markdown in their pages.

When making a PR that contains images:

1. https://github.com/calibreapp/image-actions will trigger & automatically re-size and re-encode any valid images, deleting the sources.
2. Script to replace references to the original images (ie crazy-cat.png) with the encoded images (crazy-cat.webp).

[mattaereal]

Why is this better than having users upload them wherever they want, provide direct links, and then expect the bot to return them in another comment? I want to avoid images being in our commit history.

The action you're suggesting won't work for PRs coming from forks, which is the default for any external contributor :/. Unless we start using specific PATs configured there, which I don't think is a good idea at all

I'm thinking something like:
@img-bot

link
link
link

[scode2277]

I'm currently exploring this process:

• the contributors upload the assets to a temporary online free hosting platform of our choice
• use that temporary link in the PR
• comment the links tagging the mantainer
• the mantainer then triggers the bot to do the upload with a comment, after reviewing the links and images

This is one of the best solutions for two main reasons (plus 1):

1. security: the mantainer checks the links and then approve the bot to run the upload
2. secrets availability: in this way, the secrets are available in PRs from forks, as the workflow is triggered by a maintainer and it runs in the context of the main repo
3. we dont have assets stored on github + github history

wdyt?

[DicksonWu654]

I'm currently exploring this process: -the contributors upload the assets to a temporary online free hosting platform of our choice -use that temporary link in the PR -comment the links tagging the mantainer -the mantainer then triggers the bot to do the upload with a comment, after reviewing the links and images

This is one of the best solutions for two main reasons (plus 1):

1. security: the mantainer checks the links and then approve the bot to run the upload
2. secrets availability: in this way, the secrets are available in PRs from forks, as the workflow is triggered by a maintainer and it runs in the context of the main repo
3. we dont have assets stored on github + github history

wdyt?

I think this is the best solution to both prevent images populating the commit history & preventing anyone from uploading random stuff to the s3 bucket.

The overhead for a maintainer is very small imo - it's just reviewing it to make sure it's not a troll then approving

Maybe we should use IPFS cause we're crypto :)