Adding dynamic support for upstreams without range support#224
Open
fhibler wants to merge 11 commits into
Open
Adding dynamic support for upstreams without range support#224fhibler wants to merge 11 commits into
fhibler wants to merge 11 commits into
Conversation
Integrates upstream commit ca301bd by regix1: lancachenet@ca301bd A background supervisor service (noslice-detector.sh) monitors nginx error logs for "invalid range in slice response" failures. After NOSLICE_THRESHOLD hits (default: 3), the offending host is added to a blocklist and routed through a dedicated no-slice location (slice 0) that caches without byte-range requests. The blocklist persists at /data/noslice-hosts.map across restarts. New env vars: NOSLICE_FALLBACK=true — enable/disable automatic detection NOSLICE_THRESHOLD=3 — failures before a host is blocklisted Also carries in prior commits from the upstream fork that were absent from this repo: NGINX_PROXY_*_TIMEOUT vars, NGINX_LOG_TO_STDOUT, stdout logging config in 10_setup.sh, and the multi-arch + env-var README sections. Co-authored-by: regix1 <dateague627@gmail.com>
Replaces the old CircleCI pipeline with four GitHub Actions workflows
adapted from lancache-manager:
docker-build-branch — push to any non-master branch → build+push
dev-<branch> multi-arch image to GHCR
docker-build-dev — push to master → build+push :dev image to GHCR
docker-build-release — triggered after Create Release succeeds →
build+push :latest/:release/:VERSION images
create-release — manual dispatch: validate version, create
annotated git tag, push, open GitHub Release
Adaptations from lancache-manager:
- Dropped lint workflow (no frontend/backend to lint)
- Added goss test job (install goss + run-tests.sh) before every build,
mirroring what CircleCI ran
- Removed VERSION file handling; version is derived from the git tag via
`git describe --tags` in the release workflow
- Removed build-args VERSION (Dockerfile has no ARG VERSION)
- Removed debug steps (Verify Docker authentication, Debug workflow trigger)
- Default branch is master throughout (not main)
- create-release pushes tag only, no VERSION file commit
run-tests.sh calls dgoss-tests.sh which expects lancachenet/monolithic:goss-test to already exist. Added a 'docker build' step to all three test jobs.
30_maps.conf unconditionally includes /data/noslice-hosts.map inside its map block, so if the file is absent nginx fails to parse the map and reports 'unknown noslice_host variable' for every config that references it. The file creation was previously guarded by NOSLICE_FALLBACK=true, meaning a fresh container with NOSLICE_FALLBACK=false (or any first boot before the data volume is populated) would always fail the nginx config check. Move the initialisation step outside the NOSLICE_FALLBACK block so the file is always present when nginx starts.
Ports the noslice-relevant parts of regix1's upstream fix: regix1@be68fa0 The real cause of 'unknown noslice_host variable': 15_generate_maps.sh overwrites conf.d/30_maps.conf at every startup, discarding the static $noslice_host map block that was defined there. - 15_generate_maps.sh now appends the $noslice_host map block to its generated output and writes to maps.d/30_maps.conf instead of conf.d/30_maps.conf, so both maps are always generated together. - nginx.conf: add include for maps.d/*.conf. - Remove static conf.d/30_maps.conf (fully replaced by the generated file). - Add 17_validate_generated_maps.sh which asserts both $cacheidentifier and $noslice_host are present in the generated map file before nginx starts. The /data/noslice-hosts.map path is kept (vs regix1's /etc/nginx/conf.d/) so the blocklist survives container restarts via the data volume. Co-authored-by: regix1 <dateague627@gmail.com>
References conf.d/30_maps.conf which no longer exists; the include is now in the generated maps.d/30_maps.conf produced by 15_generate_maps.sh.
Drop the matrix, create-manifest jobs, and all arm64/multi-arch plumbing. Tags are now pushed directly without a per-arch suffix.
rewrite ^ @NoSlice last; does not target named locations — nginx treats the replacement as a literal URI and tries to serve the file '@NoSlice' from disk (open() failed: No such file or directory). The correct pattern for conditionally routing to a named location is: error_page 418 = @NoSlice; if ($noslice_host = 1) { return 418; }
Cherry-picked from regix1/monolithic@2e869e8: regix1@2e869e8 Encapsulates the blocklist reset into a single script instead of a long inline one-liner. Clears /data/noslice-state.json, strips host entries from /data/noslice-hosts.map (keeping the header), then reloads nginx. Usage: docker exec lancache-monolithic /scripts/reset-noslice.sh Co-authored-by: regix1 <dateague627@gmail.com>
…hic@b88f497 Ported from regix1's commit: regix1@b88f497 20_cache.conf: - proxy_cache_lock_timeout 1h → 5m: 1h is an unreasonable safety net; 5m still covers ~27 Kbps connections while releasing stuck locks sooner - proxy_cache_valid 500 502 503 504 0: never cache server errors (defense-in-depth, lancachenet#222) 40_redirect_proxy.conf: - Forward Range header to redirect target so CDN returns 206 instead of 200 — the slice module requires a partial response to work correctly (lancachenet#175, lancachenet#207) - Add HTTP/1.1 + Connection: '' + Host header for keepalive compatibility Co-authored-by: regix1 <dateague627@gmail.com>
Files at /data/noslice-hosts.map and /data/noslice-state.json were stored directly under /data/, which is not a declared Docker VOLUME. On container recreation (image update, docker compose down && up) the writable layer is wiped, resetting the blocklist silently. Move both files to /data/cache/ — always explicitly mounted by users — so the blocklist genuinely survives restarts and container recreation. Updated: 10_setup.sh, 15_generate_maps.sh, noslice-detector.sh, reset-noslice.sh, 15_noslice.conf.disabled (comment), README.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.